[01:01:08] !log update process-control to 55529ea12d9f16c259fa3 - threshold for $500+ dedupe was too low [01:01:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:08:57] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi search: On Hold email addresses search not working? - https://phabricator.wikimedia.org/T182133#3814932 (Eileenmcnaughton) I believe this is because something is blocking the bounces from being fetched No... [01:31:31] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-CiviCRM-dedupe-FY2017/18, Patch-For-Review: Merge on addresses for MG - https://phabricator.wikimedia.org/T181088#3779205 (Eileenmcnaughton) I just ran this with the new merge rule & $500+ now has 150... [03:52:18] (CR) Eileen: "I tried to replicate this in a unit test but failed" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/395566 (https://phabricator.wikimedia.org/T181934) (owner: Ejegg) [06:55:09] PROBLEM - check_procs on frdev1001 is CRITICAL: PROCS CRITICAL: 1297 processes [07:05:06] PROBLEM - check_procs on frdev1001 is CRITICAL: PROCS CRITICAL: 1211 processes [07:10:06] RECOVERY - check_procs on frdev1001 is OK: PROCS OK: 263 processes [07:16:33] Fundraising Sprint Vaporwerewolf, Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: New Civi Import - https://phabricator.wikimedia.org/T172423#3815147 (Eileenmcnaughton) Discussed with Jerry today - I need to look at adding support fo... [12:57:48] Hi AndyRussG, there was another gap in Pivot data this morning 3-11 UTC. Do you want to check with analytics to see if it's something systematic? [15:04:18] Jeff_Green, is there a reason why running a manifest locally using puppet apply would ignore the contents of init.pp of that module? [15:04:45] is init.php only run on certain events? [15:15:29] jgleeson: i'm not sure what you mean by running it locally [15:16:00] puppet apply --verbose --debug --modulepath /vagrant/puppet/modules /vagrant/puppet/modules/smashpig [15:16:03] on the vagrant vm [15:16:12] ohhh vagrant. I've avoided that like the plague [15:16:22] ah [15:16:31] i tried it and it didn't suit my purposes, so I punted [15:17:00] other fr-tech folks would know more, sorry I can't be more helpful :-( [15:17:16] I'm really new to puppet so figuring it out as I go along. I'm trying to run the content of exec{} [15:17:41] ok no problem, I'm slowly progressing with google and the puppet coding guide [15:18:22] cool, also sorry about puppet altogether... [15:18:34] i ran mw-vagrant for quite awhile [15:18:49] okay maybe you can help with a general puppet question [15:19:21] why would someone have blocks in a manifest which are not obviously called within the same manifest via the require chain? [15:19:52] jgleeson: can you give an example or post the manifest in question somewhere? [15:20:14] puppet/modules/smashpig/manifests/init.pp [15:20:36] puppet/modules/smashpig/manifests/init.pp:30 [15:20:48] what's the gerrit repo for that one? [15:20:53] https://github.com/wikimedia/mediawiki-vagrant/tree/master/puppet/modules/smashpig [15:20:59] thx [15:21:09] thanks cwd [15:21:25] I get 'not found' [15:21:29] I'm struggling to see where line 54 gets called [15:21:36] https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/smashpig/manifests/init.pp [15:21:57] apache::site ? [15:21:59] yes [15:22:18] https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/apache/manifests/site.pp [15:22:37] basically it's calling that like a function [15:22:56] in a quirky, puppet way [15:23:05] as part of the call to smashpig/init.php ? [15:23:23] sorry [15:23:25] init.pp [15:23:26] :) [15:23:38] yep [15:23:54] hmm that doesn't seem to be happening for some reason [15:24:22] as one of the required file blocks, File['/etc/smashpig/main.yaml'], [15:24:25] is never created [15:24:56] hrm, well i cannot vouch for the workingness of any of this :) [15:25:13] but i would draw the same conclusion as you [15:25:26] you don't see any errors when puppet runs? [15:26:08] jgleeson: the require is probably there to trick puppet to behave in a declarative fashion, to make sure whatever other manifest adds that file runs before this section [15:26:10] do you have a minute to jump on a call? I wanted to explain better what it is I'm trying to do [15:27:41] cwd, I don't see any errors when adding --debug or --verbose [15:28:11] in fact, I don't see anything relating to the contents of that file when running the module [15:28:22] it's acting as if nothing is being done [15:28:30] i am just about to eat breakfast but am happy to help async [15:29:05] no problem, I'll deliberately break the code I was to check whether or not is being run [15:29:06] you see the module run though? [15:29:13] yes [15:29:19] I want to check* [15:30:31] i see it requires Class['::apache::mod::rewrite'] [15:31:48] jgleeson: notify, like "notify { 'this thing just ran': }", is handy for debugging [15:32:03] it just prints messages at run time [15:32:43] if I break the exec stanza, no errors are reported. If I add random text to the class, I get a syntax error during the run [15:33:02] which smells like exec is not being run automatically when I use puppet apply [15:33:15] maybe that's my flawed assumption [15:33:32] you mean the smashpig schema part? [15:33:35] that's not how it behaves in production, but it could be that vagrant somehow overrides normal puppet behavior [15:33:55] I've took vagrant out of the mix for the time being [15:34:02] I'm on the vm itself calling puppet apply [15:34:13] which technically should skip the vagrant abstraction [15:35:05] ok [15:37:07] i would hack the apache::site class to dump some information [15:47:30] fixe [15:47:34] d [15:47:42] it was a permissions issue [15:47:53] thanks for the help cwd Jeff_Green [15:48:08] great! [15:49:58] classic [15:52:43] yeah the directory the file resource was trying to write to didn't exist so gonna work out how to add mkdir -p like behaviour to the resource [15:52:46] pcoombe: ok! yesterday IIRC it wasn't an actual problem, just some maintenance downtime [15:53:08] fr-tech morning! [15:53:22] Jeff_Green: cwd; hi! :) [15:53:29] morning AndyRussG [15:53:37] morning AndyRussG :) [15:53:58] pcoombe: morning, too :) (though it's your afternoon :) ) [15:55:45] orale vato, wass happenin [15:57:38] cwd: podría estar peor, jejeje [16:02:07] muy bien! [16:17:22] all fixes applied. Now to destroy this instance and vagrant up from scratch (crosses fingers) [16:18:05] cwd: :) [16:23:14] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review: CentralNotice: instrument debugging for "Large banner limit and switch" mixin - https://phabricator.wikimedia.org/T180478#3816522 (AndyRussG) Open>Resolved a:AndyRussG [16:46:57] cwd did you ever work on dash? [16:47:14] mepps: a little yeah [16:47:27] it keeps maxing out my mysql connections--did you ever run into that? [16:52:11] cwd ^ [16:52:47] mepps: not that i can remember, do you know if your setting is the same as prod? [16:53:16] hmm i do not cwd, i guess you would have been on vagrant, huh? [16:54:04] yeah i believe so [16:54:16] i can find the prod setting, one sec [16:55:01] looks like the default is 151 [16:55:05] weird [16:59:01] pcoombe: seems this time it's some problem with the Pivot job, I'm afraid. Analytics is working on it [16:59:45] hmm i have the same setting locally [17:00:43] sounds like a rogue db connector [17:03:09] or a loop of them [17:08:08] AndyRussG: ok, thanks for checking [17:17:07] pcoombe: likewise thx! :) [17:21:25] PROBLEM - check_swap on civi1001 is CRITICAL: SWAP CRITICAL - 49% free (3665 MB out of 7623 MB) [17:25:06] PROBLEM - check_swap on civi1001 is CRITICAL: SWAP CRITICAL - 81% free (6141 MB out of 7623 MB) [17:30:03] ACKNOWLEDGEMENT - check_swap on civi1001 is CRITICAL: SWAP CRITICAL - 81% free (6141 MB out of 7623 MB) Jeff_Green this is just the residual swap high watermark, kernel reports 10GB RAM free [17:36:12] fr-tech did somebody get a chance to kill that 'running job list' job? [18:27:33] ejegg|afk, no, should i do that? [18:29:41] XenoRyet do you know anything about this ^^ [18:33:38] mepps: No, not sure what that's about. First I've heard of it. [18:33:57] I've got Scrum of Scrums in like 2 mins though. [18:34:24] Speaking of: fr-tech got anything for SoS? [18:34:38] XenoRyet: thanks, nothing here :) [18:34:53] cool cool [19:25:15] PROBLEM - check_procs on frdb1001 is CRITICAL: PROCS CRITICAL: 1031 processes [19:25:47] uuuh [19:28:59] lot of queries alright [19:42:36] civi lag is 2 days currently... [19:42:50] cwd: I'm hearing some people say that civi is running a bit slow. Could that be related to your last comment? [19:43:20] ah ccogdill what do you mean the lag it 2 days? [19:43:31] according to the dash widget [19:43:46] avg age of last 10 donations [19:43:56] ah so it is [19:44:23] have we had any recent dash patches? [19:44:43] since I'm pretty sure that even here Dec6 is not 2 days ago.... [19:45:04] mepps: was working on some things [19:45:15] PROBLEM - check_procs on frdb1001 is CRITICAL: PROCS CRITICAL: 1165 processes [19:45:35] I sent emails out 15 mins ago that are returning click data but 0 donations [19:45:39] so something's up [19:46:12] there is a query I'm gonna kill [19:46:27] TY for flagging ccogdill, can confirm banners look slow too [19:47:23] cc MBeat ^ [19:47:59] ty ccogdill [19:48:32] I just killed a query by user 181 [19:49:36] geez user 181 [19:49:41] ;) [19:49:55] lag down to 20 mins according to dash [19:50:15] RECOVERY - check_procs on frdb1001 is OK: PROCS OK: 285 processes [19:51:02] That tracks with what grafana says right now. [19:52:44] XenoRyet: do we use max(receive_date) or the receive_date of the latest id to calc lag? [19:54:39] dstrine: yep probably [19:55:24] dstrine my dash patch hasn't come out of WIP yet [19:56:40] Speaking of grafana, there is a weird dip in the 'donations queued to civicrm' graph from about 19:25 to 19:50 [19:56:55] Looks like it's saying the queues were empty, but that can't be right. [19:57:27] cwd do you know anything about the running job list job? [19:57:48] i do not, i assume it is a process control job? [19:58:15] i guess so, ejegg|afk was asking whether we'd killed it but i don't know why [19:59:05] mepps: cwd - no it was a user-initiated query [19:59:29] (if you mean the slow query that I killed) [19:59:32] yeah i saw that [19:59:53] it seems like the UI is more wont to create a runaway query than the cron job [20:02:44] I'm going to get her to show me on screen what happened after her meeting [20:03:53] no eileen this was something different that ejegg|afk asked about [20:04:10] unless that was the same thing? [20:04:34] No, wouldn't be the same, I don't think. [20:06:38] (PS14) Mepps: Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [20:07:45] do we know why we got all these failmails around thank_you_send earlier today fr-tech? [20:08:18] That might be what ejegg|afk was talking about. Maybe that job had to do with that one? [20:09:22] (CR) jerkins-bot: [V: -1] Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) (owner: Mepps) [20:09:23] eileen: I just forwarded an email from Danny to fr-tech. I don't think it's related to what is being discussed here but I'm wondering if you have feedback subject" Fwd: WMF - large donation: $1000" [20:10:36] hmm - I feel like ejegg|afk sent an email about that yesterday [20:11:04] oh I see- from staging - hmm [20:11:56] yeah that's weird right? [20:12:34] (PS15) Mepps: Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [20:13:01] I don't know why these old donations are triggering emails but in general we shouldn't trust any of the staging emails right? This wouldn't indicate any weirdness on prod right? [20:13:22] (CR) jerkins-bot: [V: -1] Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) (owner: Mepps) [20:14:54] (PS16) Mepps: Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [20:15:47] (CR) jerkins-bot: [V: -1] Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) (owner: Mepps) [20:18:14] (PS17) Mepps: Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [20:19:11] (CR) jerkins-bot: [V: -1] Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) (owner: Mepps) [20:22:37] (PS18) Mepps: Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [20:25:59] (PS19) Mepps: Urls direct to boards [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/394664 (https://phabricator.wikimedia.org/T120000) [20:43:36] dstrine: I'm just wondering if they got triggered when I asked Jkim to test her stripe import on staging [20:43:45] ah [20:43:46] I couldn't think of a reason - but that could be it [20:43:50] maybe [20:44:25] either way we shouldn't be trusting staging data or emails [20:44:28] right? [20:44:33] yep - it's a stripe one & yep [20:44:38] ok cool [20:45:38] I guess we shouldn't turn off staging email cause we want to be able to test them? [20:47:58] I just set all threshholds to 5000000.00 [20:47:59] but - that won't survive our next DB restore [20:48:28] so we can either say 'hey it'll be alright through BE & I may not hurt us again so that's cool' or 'hey it's ok for now, let's log a phab for later' [20:53:01] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Consider finding a way to ensure MG large donation emails (& others?) don't get sent from staging - https://phabricator.wikimedia.org/T182244#3817886 (Eileenmcnaughton) [20:53:06] created https://phabricator.wikimedia.org/T182244 [21:00:06] RECOVERY - check_swap on civi1001 is OK: SWAP OK - 100% free (7548 MB out of 7623 MB) [21:16:00] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Possibility to create bad query from dedupe by fishing net rule - https://phabricator.wikimedia.org/T182251#3818010 (Eileenmcnaughton) [21:17:21] Jeff_Green: cwd ^^ is follow up on that slow query [21:50:09] (PS1) Eileen: Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/395858 (https://phabricator.wikimedia.org/T18108) [21:54:39] XenoRyet finally got the build passing on my dash patch if you want to take a look: https://gerrit.wikimedia.org/r/#/c/394664/ [21:56:44] cwd Jeff_Green - am turning on one more dedupe job - on orgs - per T181089 [21:56:45] T181089: Let dedupe run on orgs - https://phabricator.wikimedia.org/T181089 [21:57:11] - I guess ideally someone would check commit in process_control before deploy [21:57:14] also fr-tech i'm still not sure what's up with the run job listing that ejegg|afk mentioned, i don't want to turn something off if i don't know what it does or why! [21:58:07] I'm kind of in the same boat there. [21:58:20] oops, always forgetting to change my nick back after lunch. [22:00:25] what's in the run-job listiing [22:09:57] eileen: ok [22:16:01] (CR) Mepps: [C: 2] Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/395858 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [22:16:35] (CR) XenoRyet: [C: 2] Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/395858 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [22:20:06] (Merged) jenkins-bot: Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/395858 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:00:31] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi search: On Hold email addresses search not working? - https://phabricator.wikimedia.org/T182133#3818364 (Eileenmcnaughton) This is a duplicate of T179547 It should be OK now - a single email was blocking e... [23:01:43] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi search: On Hold email addresses search not working? - https://phabricator.wikimedia.org/T182133#3818373 (Eileenmcnaughton) [23:01:45] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: process unsubscribes job is failing - https://phabricator.wikimedia.org/T179547#3728244 (Eileenmcnaughton) [23:04:56] !log update process-control to b1cd5150e8ef2bf9e4212 (reenable bounce processing, enable dedupe on orgs) [23:05:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:06:38] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: process unsubscribes job is failing - https://phabricator.wikimedia.org/T179547#3818397 (Eileenmcnaughton) We have removed the one email that was blocking & I have re-enabled the job. Am still reviewing @Ejegg p... [23:10:02] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: process unsubscribes job is failing - https://phabricator.wikimedia.org/T179547#3728244 (Eileenmcnaughton) [23:10:17] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Civimail bounce processing failing - https://phabricator.wikimedia.org/T181934#3818428 (Eileenmcnaughton) [23:13:31] (PS1) Eileen: Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) [23:13:50] (CR) Eileen: [C: 2] Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:13:58] (CR) jerkins-bot: [V: -1] Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:14:21] (CR) jerkins-bot: [V: -1] Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:15:05] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi search: On Hold email addresses search not working? - https://phabricator.wikimedia.org/T182133#3818436 (MBeat33) That is great news, @Eileenmcnaughton thank you! It's working perfectly now. [23:39:50] (CR) Eileen: [C: 2] "recheck" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:40:15] (CR) jerkins-bot: [V: -1] Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:41:12] (Merged) jenkins-bot: Allow Organization merge to resolve casing on organization_name [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/395882 (https://phabricator.wikimedia.org/T18108) (owner: Eileen) [23:45:28] !log update CiviCRM from 85263f625763a454b2664062154c8b791e5a6838 to 2db9c761884c1c48f4e1dd41be416e029b23b481 (improve org merges) [23:45:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:52:52] brb, gotta do some dad-type stuff for a few.