[00:09:40] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3438093 (AndyRussG) Thanks!!! There are results for a search for the banner names in LogStash: [[ https://logstash.wikimedia.or... [00:27:12] * awight squeezes eyes shut. I swear I didn't see anything on the private channel :) [02:34:32] (CR) Jforrester: [C: 2] build: Updating mediawiki/mediawiki-codesniffer to 0.10.0 [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/364013 (owner: Umherirrender) [02:36:28] (Merged) jenkins-bot: build: Updating mediawiki/mediawiki-codesniffer to 0.10.0 [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/364013 (owner: Umherirrender) [08:13:26] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3438513 (jcrespo) The queries seem to be writing 113800 rows in a single query. I cannot help you with that, that is not a server... [14:54:24] (PS10) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [14:55:27] (CR) jerkins-bot: [V: -1] WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) (owner: Mepps) [15:21:35] (PS11) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [15:22:41] (CR) jerkins-bot: [V: -1] WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) (owner: Mepps) [15:35:05] PROBLEM - check_nginx on payments1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name nginx [15:35:06] PROBLEM - check_payments_wiki on payments1001 is CRITICAL: connect to address payments1001.frack.eqiad.wmnet and port 443: Connection refused [15:35:06] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[nginx],Package[nginx-full] [15:35:07] PROBLEM - check_ssl on payments1001 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused [15:37:55] ACKNOWLEDGEMENT - check_nginx on payments1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name nginx Jeff_Green nginx/libssl package update snafu, working on it... [15:37:57] ACKNOWLEDGEMENT - check_payments_wiki on payments1001 is CRITICAL: connect to address payments1001.frack.eqiad.wmnet and port 443: Connection refused Jeff_Green nginx/libssl package update snafu, working on it... [15:37:57] ACKNOWLEDGEMENT - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[nginx],Package[nginx-full] Jeff_Green nginx/libssl package update snafu, working on it... [15:37:57] ACKNOWLEDGEMENT - check_ssl on payments1001 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused Jeff_Green nginx/libssl package update snafu, working on it... [15:47:21] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3439467 (AndyRussG) >>! In T170591#3438513, @jcrespo wrote: > The queries seem to be writing 113800 rows in a single query. Ouch!... [15:54:25] Morning dstrine :) looks like T170591 is a doozie. 99% certain that it's not due to our code directly, but we ought to help investigate, at least initially. Apparently causing badness on prod [15:54:25] T170591: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591 [15:54:57] AndyRussG: yeah I'm just reading your notes on the task [15:55:14] dstrine: cool! Yeah just sent another comment [15:55:15] RECOVERY - check_nginx on payments1001 is OK: PROCS OK: 49 processes with command name nginx [15:55:15] RECOVERY - check_payments_wiki on payments1001 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.053 second response time [15:55:16] RECOVERY - check_ssl on payments1001 is OK: SSL OK - Certificate payments.wikimedia.org valid until 2018-10-25 23:59:59 +0000 (expires in 468 days) [15:55:32] katie is talking with the rest of tech on tuesday morning. She can bring up cross team stuff and bump priority [15:55:32] (PS7) Mepps: Curl Wrapper fix and use provider config [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/365012 [15:56:03] so if we have a specific ask from another team we should make that clear [15:56:04] dstrine: K we'll see how it plays out, I guess. I think Jaime's comment will raise some red flags [15:57:21] In fact the main pain points so far are in the prod/performance domain, it should be most urgent for the folks responsible for that infrastructure [16:00:15] RECOVERY - check_puppetrun on payments1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:04:42] also pinging some folks on IRC [16:06:05] AndyRussG: thanks! let me know how the response is and we can escalate if needed [16:06:30] dstrine: K will do, thx :) [16:08:05] Fundraising Sprint Judgement Suspenders, Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising-Backlog, and 2 others: CentralNotice banner sequence: unit tests - https://phabricator.wikimedia.org/T144456#3439526 (DStrine) [16:08:42] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising Sprint Far Beer, and 12 others: CentralNotice banner sequence: implement feature for MVP - https://phabricator.wikimedia.org/T144453#3439527 (DStrine) [16:08:54] (PS12) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [16:27:24] ejegg: I'm trying to remember something you said.. did you mention that some of the silverpop/civi stuff is on prod already? [16:27:51] dstrine: yep! [16:27:59] ok cool [16:28:00] I don't think the UI is visible [16:28:04] ok [16:28:13] and I don't know if eileen set up the jobs to actually fetch the data [16:28:21] but the code itself is up :) [16:30:13] cool [16:30:37] AndyRussG: should we make this a "high" priority or "unbreak now"? T170591 [16:30:38] T170591: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591 [16:32:51] (CR) Ejegg: [C: -1] "So close! header_size is still not quite working with real responses." (4 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/365012 (owner: Mepps) [16:36:11] dstrine: yeah let's unbreak-now it [16:36:22] (unbreak-it now?) [16:36:27] ok cool [16:36:42] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3439587 (DStrine) p:Triage>Unbreak! [16:37:03] I want to make sure this conveys the right amount of urgency or others cc'd [16:37:31] yep makes sense :) [16:43:32] dstrine: I'm gonna add some other projects too to see if that helps [16:43:39] cool [16:50:01] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Performance, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3439599 (AndyRussG) [16:51:03] K just got some replies on -operations... Apparently the people who could help get more info are off for the weekend, because the earth is round [16:51:33] However also got the excellent (and obvious) suggestion of trying to reproduce locally [17:20:22] Fundraising Dash, Fundraising-Backlog, fundraising-tech-ops: Dash should run on a low port - https://phabricator.wikimedia.org/T170697#3439700 (cwdent) [17:24:58] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Performance, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3439737 (jcrespo) > Is there a log of the queries somewhere? This is what I got from the log yo... [17:34:06] (PS4) Ejegg: Do not add id if the field does not exist [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/364362 (owner: Eileen) [17:34:39] (CR) Ejegg: [C: 2] "Ah, looks like we need this for the new mailing data table. Seems safe!" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/364362 (owner: Eileen) [17:39:50] Hey MBeat dstrine: just a heads up that we won't be doing a Brazil pre-test today as I haven't been able to track down all the translations. Going to reschedule for next week. [17:40:07] ok cool pcoombe [17:40:11] totallly fine pcoombe [17:46:10] (PS8) Mepps: Curl Wrapper fix and use provider config [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/365012 [17:48:36] (Merged) jenkins-bot: Do not add id if the field does not exist [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/364362 (owner: Eileen) [17:49:01] okay ejegg got another patch up [17:50:10] although wait [17:50:30] you said headersize wasn't working with live curl's? let me look at that [18:02:22] mepps just the offset [18:02:29] that +1 was breaking live stuff [18:05:34] Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Add ability for MG to import to Primary address type - https://phabricator.wikimedia.org/T169025#3439804 (LeanneS) @Ejegg and @Eileenmcnaughton I... [18:11:42] do you mean the header_size + 1 ejegg? [18:12:42] yep, the +1 is the issue [18:17:19] oh okay cool [18:25:41] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Operations, Performance, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3439891 (AndyRussG) Ah OK @jcrespo... Yeah, I see it was right there, sorry... And thanks!!! [18:31:41] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3439895 (AndyRussG) [18:35:33] dstrine: update about T170591 ... So now it seems it's not a core issue, probably unrelated to the old banners slow-to-save... Rather it's related to Translate wiki. So that puts it on our plate a little more--we have to at least check that it's not something bad about how CN is using Translate extension, I think [18:35:33] T170591: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591 [18:46:55] (CR) Ejegg: [C: 2] "Looks good!" (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/365012 (owner: Mepps) [18:47:05] ejegg yay! [18:47:52] (Merged) jenkins-bot: Curl Wrapper fix and use provider config [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/365012 (owner: Mepps) [18:50:30] yay indeed! [18:50:43] looks like the following patch is about ready to shed its WIP too [18:52:34] woo! [18:56:17] did you have more changes in mind, or is it ready for review now? [18:57:24] hmm i just remembered your comment about the tests, i can look at that [18:59:18] cool [19:04:10] mepps: and ejegg|afk check out https://phabricator.wikimedia.org/project/board/2894/ [19:04:27] #fr-ingenico-integration_17_18 [19:04:48] this should help organize the current ingenico work [19:06:13] cool! i'll start tagging some stuff when im back at a desk [19:31:01] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Adding translatable message to banner FR2015_translations takes a long time and shows error - https://phabricator.wikimedia.org/T167442#3333075 (AndyRussG) Searching LogStash for the [[ https://logstash.wikimedia... [19:44:36] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, WMDE-Fundraising-CN: WMDE banners failing to save - Timing out on save - https://phabricator.wikimedia.org/T170591#3440077 (AndyRussG) Maybe the same bug as is causing T168926? [19:44:45] (PS13) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [19:44:51] take a look now ejegg [19:44:53] (CR) jerkins-bot: [V: -1] WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) (owner: Mepps) [19:45:01] ooh nevermind [19:46:26] :) [19:47:33] it's just a rebase ejegg [19:47:39] ejegg standup? [19:48:27] fr-tech bandwidth just got really slow here, trying to get in the hangout [19:53:26] fr-tech this might slay our paypal dead session problems: https://gerrit.wikimedia.org/r/364825 [19:55:03] (PS14) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [20:03:40] (PS2) Mepps: PayPal EC: use ajax to get redirect URL [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/364825 (https://phabricator.wikimedia.org/T167923) (owner: Ejegg) [20:03:42] (CR) Mepps: [C: 2] PayPal EC: use ajax to get redirect URL [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/364825 (https://phabricator.wikimedia.org/T167923) (owner: Ejegg) [20:06:00] (Merged) jenkins-bot: PayPal EC: use ajax to get redirect URL [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/364825 (https://phabricator.wikimedia.org/T167923) (owner: Ejegg) [20:08:32] (CR) Ejegg: [C: -1] "Looks like this should work! Just some quibbles with a default and how we're ensuring we have a subdomain." (5 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) (owner: Mepps) [20:51:17] hi eileen ! [20:51:39] Leanne has been trying out your 'primary location' patch [20:51:55] and she says the contact ID has disappeared as a mappable field [20:52:00] but it doesn't look like your patch did that [20:52:11] Guessing it was the version upgrade? [20:57:26] (PS15) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [20:58:08] ejegg updated patch ^^ [20:58:17] (CR) jerkins-bot: [V: -1] WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) (owner: Mepps) [20:58:24] think i caught it all but a little distracted this evening due to crying baby, oh nevermind [20:59:28] hope he cheers up soon! [20:59:36] (PS16) Mepps: WIP: Implement HostedCheckout [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) [21:05:56] (CR) Ejegg: "tiny tweaks requested" (3 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/363458 (https://phabricator.wikimedia.org/T163946) (owner: Mepps) [21:08:55] Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Add ability for MG to import to Primary address type - https://phabricator.wikimedia.org/T169025#3440379 (LeanneS) I attempted testing on the prod... [21:18:56] (PS4) Ejegg: Enable PHP_CodeSniffer, fix complaints [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285228 (owner: Awight) [21:19:50] (CR) jerkins-bot: [V: -1] Enable PHP_CodeSniffer, fix complaints [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285228 (owner: Awight) [21:29:51] (PS5) Ejegg: Enable PHP_CodeSniffer, fix complaints [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285228 (owner: Awight) [21:31:19] (CR) jerkins-bot: [V: -1] Enable PHP_CodeSniffer, fix complaints [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285228 (owner: Awight) [21:35:21] Fundraising Sprint Loose Lego Carpeting, Fundraising Sprint Murphy's Lawyer, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Add ability for MG to import to Primary address type - https://phabricator.wikimedia.org/T169025#3440437 (LeanneS) Also, when creating the new map... [22:15:57] have a great weekend, all! [22:28:16] (CR) Legoktm: [C: -1] Enable PHP_CodeSniffer, fix complaints (3 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/285228 (owner: Awight) [23:30:26] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/364363 (owner: Eileen) [23:36:59] (CR) jerkins-bot: [V: -1] Omnimail, enable view tab into mailing events. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/364363 (owner: Eileen)