[17:18:02] Jeff_Green: ejegg: sorry to harp on this... did one of u fix the thank-you job? [17:18:37] not I [17:19:03] HMM [17:19:11] exactly [17:19:18] nor I [17:19:27] ooboy ok thanks [17:19:36] ghost in the machine [17:19:38] I'll dig in [17:24:32] Jeff_Green: just... a minor complaint about the current version of Jenkins. I'm on my fifth login attempt, it won't log me in with the right password, and doesn't display any error if I use an incorrect password [17:24:57] * awight glares at Ubuntu maintainers [17:25:01] hmmm [17:25:32] and you think a version that's 10% newer will be more than 0.3% good overall? [17:25:43] cron wouldn't punish you for being a java hater [17:25:53] well you have to subtract the 50% more broken shite [17:26:11] want me to kick it? [17:26:17] But. There isn't a PPA or deb repo or whatever for WMF jenkins [17:26:18] ? [17:26:37] nah I will inject it with some fatal NPE once I login successfully [17:26:56] i dunno re wmf repo [17:27:05] Jeff_Green: ok all it took was threats. Logged in now. [17:27:58] This is not exactly an active project, but: https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/jenkins-debian-glue [17:28:33] gosh I want to run out and get it [17:28:41] :D [17:28:46] here's a dollar. [17:28:53] bring me tha change [17:29:03] ICE CREEEEAM! [17:29:19] fwiw it's been a long time since i saw my jenkins-faceplanted-so-restart-it cron have to restart it [17:29:30] that's better than the aluminium/grosley days [17:30:28] so, https://integration.wikimedia.org/ci/ is using 1.565.3 [17:30:33] iono how they do that. [17:31:31] we have 1.424.6 [17:31:45] I guess they only do palindrome minor versions [17:32:12] :-P [17:32:40] some kind of inflation strategy [17:32:46] which project was it converges to pi? [17:32:49] --yeah we've been really busy developing the next number [17:32:52] ejegg: tex i think [17:32:56] super arrogant [17:33:09] literate programming :) [17:33:34] I think there's some other stuff which is always adding .999's [17:33:36] grr [17:33:40] ok back to the grindstone [17:33:42] makes sense if the changes are smaller each time. Getting closer to perfection. [17:34:21] that's... what is annoying [17:34:27] heh [17:34:37] I like a project such as Sagrada Familia [17:40:00] * AndyRussG waves [17:40:07] Hi awight ejegg Jeff_Green atgo [17:40:13] Hi AndyRussG [17:40:14] howdy! [17:40:18] hey :) [17:40:19] AndyRussG: mornin [17:40:20] sagrada familia! amazing!! [17:40:26] hey AndyRussG [17:40:30] :) [17:40:31] sorry we don't actually have a virtual water cooler [17:40:36] atgo: yes, I want to see [17:40:37] I was just working on getting your accounts set up [17:40:49] awight you will not be disappointed [17:41:01] anyway. hi AndyRussG! [17:41:14] and ejegg and awight and Jeff_Green and whoeveer else is actually here :P [17:41:18] hi :) [17:41:28] * AndyRussG looks up sagrada familia [17:41:45] atgo: I added a bunch of cards for the bucket work, BTW, I guess u saw 'em :) [17:41:56] yeah.. working through those [17:41:58] There was a cathedral next to my school that had been in the works nearly as long. St John the Divine, i think. [17:42:01] turns out a lot happens in 2 days! [17:42:26] ejegg: niice. But with less expenditure of effort? [17:42:27] sometimes 8p that happens when it occurs [17:42:35] atgo: pls let me know if u have any questions [17:42:48] AndyRussG will do! i'm catching up... and now i'm sick. [17:43:04] so it may be slow, but i'm sure it's a good approach! [17:43:05] atgo: oh no, sorry to hear that! :/ [17:43:19] yeah I think no blockers right now [17:43:36] get better soon! [17:44:04] thanks AndyRussG! [17:44:23] np... ;) [17:44:44] so, in the long term, i'd like to try to get into the habit of including testing in the feature. but that's a bigger discussion with the team :) [17:44:59] Yes, this is a testing sort of thing. [17:45:25] Maybe that's a good thing to parallelize--lighting our pyre with the eternal flame of WMDE's banner testing infrastructure. [17:45:46] I imagine... I put it in the bucket plan since it's a significant change to important code just before the year-end FR [17:46:58] hopefully good to catch bugs in edge case [17:47:00] cases [17:47:34] yeah awight we're talking about how AndyRussG has a separate card for writing unit tests [17:48:07] in fact in the propsed steps, 2 separate steps [17:48:16] awight pizzzacat1 ejegg did we deploy the new WP forms this morning? [17:48:34] or any changes in the last 48 hours [17:48:45] step 3 would be for tests for steps 1-2, and step 7 is for tests for steps 4-6 [17:48:51] atgo: I did not do [17:49:09] no, didn't. Want me to? [17:49:18] no... thanks [17:49:22] well, eventually [17:49:22] k [17:49:23] could certainly be reorganized tho [17:49:25] i have no opinion haha [17:49:34] there's just been high rejections over the last 48 hours [17:49:42] ooh, huh [17:49:54] not sure if it's on our side or on WP... we can wait to hear from them [17:50:25] * awight mutters about how we are wasting our time playing craps with WP [17:50:28] AndyRussG yeah i think we should talk about that as a team and see if we can't some good norms [17:50:46] but... that's a bigger conversation we should maybe have at retro next week [17:50:50] atgo: cool sounds great :) [17:51:01] for sure. "will it break everything?" == needs tests :D [17:51:20] ^ solid principle [17:51:30] ha i 100% agree [17:51:33] i think we should test ALL the things [17:51:51] atgo: um no [17:51:54] eventually, yes [17:51:57] exactly [17:51:57] but it's not practical [17:52:03] atgo: I did try that with the Editor Campaigns stuff I was doing [17:52:05] bigger conversation :P [17:52:13] writing tests takes (me) about 2x as long as writing the feature [17:52:27] if we can hire 3x our team... then test everything [17:52:37] awight yeah let's definitely talk about it :) [17:52:42] I am :p [17:52:47] as scrum master, i just want our cards to make sense [17:52:50] but we sould have the whole team [17:52:58] okok [17:53:07] the main issue was that sometimes writing several features led to an early big refactor, so some tests ended up being not used (tho I learned a lot writing them) [17:53:18] ejegg did the login button change I made on patch https://gerrit.wikimedia.org/r/#/c/169236/ get somehow nonmerged? [17:53:21] i'll add it to the agenda for retro now :) [17:53:30] \o/ cool [17:54:01] or I should word that better: did it not get merged? [17:54:02] pizzzacat: It got merged, but I don't think it got gulped [17:54:18] oh, right. ok. [17:54:22] The stuff in dist is still pre-stylez [17:55:09] do you want me to make a patch that dev-ifies the dev repo Dash? aka removing gulp/dist functions? [17:55:38] then when we want to deploy, we just add a patch in the deploy branch for it? [17:56:22] All we need to do is point the static dir back to src in server.js, right? [17:56:52] that, and some changes in index.html [17:57:03] oh, what changes in index.html? [17:57:49] We should make the index.html work for dev, then add something in the gulp replace task to make any of the changes to point it back to dist. [17:58:00] wtf "gulp/dist" I like it though, I think. [17:58:48] actually ejegg I lied, that wsa fixed already [17:58:59] woohoo [17:59:18] I wonder if you fixed it or if I inadvertently committed my local fix [17:59:26] heh, i dunno [17:59:38] yeah so just swapping /src and /dist is perfect. so ez [17:59:52] ok gulp = http://gulpjs.com/ [17:59:56] * awight goes back to work [17:59:58] yerp [18:00:26] unfortunately doesn't work with the node version we have in prod, so even if we wanted to add a compile step it we couldn't [18:00:34] * awight sighs with nostalgia about having a build step for PHP cruft [18:00:39] hey ccogdill are we in the standup or retro event? [18:00:48] I'm sure its at least as simple as maven [18:00:53] hehehe [18:00:55] mvn2 [18:00:58] standup [18:01:05] but Pats and I are waiting for our room [18:01:23] ok.. just not sure since we've got a lot of google events going on [18:01:41] pizzzacat: guess we should de-lint our js and make that task voting again at some point [18:03:44] ejegg sounds good to me [18:03:46] ,hey atgo I think we should do the retro event? [18:03:50] we just got in [18:03:56] meganhernandez an di are in the other one [18:04:24] i am in the standup calendar [18:06:44] ccogdill: fyi, we had a few batches of TY letters which did not go out this morning, affecting 660 people [18:06:51] Trying to fix that now... [18:09:56] Jeff_Green: I just hosed the entire civi db... stray semicolon. [18:10:03] Jeff_Green: can you help... [18:10:22] !log disabled TY job [18:10:26] Logged the message, Master [18:10:51] Jeff_Green: I need to copy values from one column, from a recent back onto the current db [18:11:09] I can write the query, but need the backup data available as a db [18:12:02] uh [18:12:05] which civi db [18:12:20] the production one or civi one? [18:12:26] err or dev one rather [18:12:38] production [18:12:40] :-/ [18:12:43] :-( [18:12:51] that's what I'm saying [18:12:53] how fresh does the restore need to be? [18:13:04] whatever we got... [18:13:31] we should have a backup within the past 24H which will take an hour or so to load up [18:13:54] or we have a restore from last thursday at lutetium dev_civicrm [18:14:12] oh, guess I should stop the non-usd recalc job [18:14:17] I'm going to do something like, update wmf_contribution_extra e join backup.wmf_contribution_extra old_e on e.id=old_e.id set e.no_thank_you = old_e.no_thank_you; [18:14:24] ejegg: nah it's fine [18:14:34] Jeff_Green: the newer the better [18:14:41] Jeff_Green: I can't tell which rows I damaged... [18:14:51] awight: how painful is it in the meantime? [18:15:05] Jeff_Green: no problem, we just have to leave the TY job disabled [18:15:17] ccogdill: ^^ no TYs will be going out until I clean up the mess I just made... [18:15:32] I can clobber dev_civicrm with last night's run, dump just the table you need, and restore it at some new/temporary table name on db1025 [18:16:02] Jeff_Green: that sounds perfect [18:16:07] i guess we might have enough space to restore the full db on db1025, checking [18:16:18] I sorta hate the idea of doing that though [18:16:19] The single table will be good enuf [18:17:38] atgo do you know what the December online fundraising goal is for 2014? [18:17:47] in... dollars? [18:17:51] yes [18:17:55] awight: I'll start the restore then I gotta grab food [18:18:00] no, sorry [18:18:11] ok, I'll email Megan [18:19:40] Jeff_Green: thanks! Yeah it's a minor thing, if I can fix it some time today that's enuf to keep everyone happy. [18:20:12] it'll take longer than my food run to restore anyway [18:20:29] yes :) [18:21:01] I might suddenly develop the need to chat with you about unrelated things, but will resist mightily. [18:21:07] ha ok [18:21:09] started [18:21:14] back in ~20 [18:21:19] such as "error: puppetrun" :p [18:21:30] where do you see that? [18:21:39] hehe I'm just trying to give u nightmares, ignore me [18:21:44] oh [18:21:46] bad man [18:21:47] but there is a lot of nightly failmail [18:21:59] oh yes, and I pay attention to it all [18:22:07] * awight coughs [18:22:14] * awight politely [18:22:35] and then I do ; t f root and delete in one sweeping sweep [18:22:51] the only thing that's currently abnormal is aluminium failmail [18:23:05] the rest is routine package maintenance alerts and such [18:23:22] I'm just messing with you, sorry [18:23:26] we're now reliably up to date within a day for security updates [18:23:38] ok back in a bit [18:23:39] just in time for winter's zero-days! [18:38:44] back [18:39:19] ejegg|ScrmOScrms: want to buy a vowel :p [18:46:52] Jeff_Green: fwiw, the TY error came from this line, $rt = @mail($to, $this->EncodeHeader($this->SecureHeader($subject)), $body, $header, $params); [18:46:55] in PHPMailer [18:47:07] rlly [18:47:08] not at all cool, the "@mail" means we were suppressing the real error [18:47:33] But i'm mentioning the failure to u cos I have no clue why PHP's mail() function would systematically fail [18:47:43] huh [18:47:54] last I looked at php vs mail, the situation was somewhat grim [18:48:07] http://php.net/manual/en/function.mail.php [18:48:38] it says, it returns false if the mail is not accepted for delivery.. but I assume we're configured to send to localhost... [18:48:52] I seem to remember it being incapable of a different From vs envelope sender [18:49:28] php mailing is a horrible mess... but we do have the option of switching between a few backends if we ever decide that will help. [18:49:30] awight have you looked in mail logs? [18:49:40] privs... hmm lemme see [18:50:04] oh weee I'm in the right group [18:50:28] thx [18:51:04] i don't see any noqueue lines, that's something [18:52:20] Jeff_Green: check out the hole after Nov 5 14:44:01 [18:52:23] $this->EncodeHeader($this->SecureHeader($subject)) seems like the likely candidate for exploding [18:52:27] that's when the job started to fail [18:52:44] when did they spontaneously recover? [18:53:00] looking... [18:53:09] right, cos there are successful mailings from other processes [18:53:44] the first successful TY job was at Nov 5, 2014 3:48:14 PM [18:53:50] the logs are really wacky [18:53:54] first failure at Nov 5, 2014 2:34:18 PM [18:54:04] did you notice how the time jumps all over the place? [18:54:09] wat. [18:54:22] no, can u give an example? [18:54:30] lemme actually look on barium, maybe its an artifact of central log collection [18:55:31] that must be it, time is linear in barium logs [18:55:42] Wouldn't have anything to do with the recalc job, I hope. I ran that with nice 15, but it was sitting on a big chunk of ram for a while [18:56:32] ejegg|ScrmOScrms: I think that would have emitted OoMs [18:56:45] ok, that's good [18:57:05] ejegg|ScrmOScrms: the refund variance might be related, but I'm not worried about that [18:58:39] awight: what is the fail indicator, how are you telling that messages didn';t go out? [18:59:07] There's an exception from PHPMailer, e.g. in syslog, Nov 5 14:44:01 [18:59:12] oic [18:59:15] PhpMailer died unexpectedly: Could not instant [18:59:15] iate mail function.
[18:59:17] blah blah [18:59:51] every send fails... then suddenly, every send succeeds [19:00:35] the next line in syslog is even more concerning, error sending email from fr-tech to fr-tech [19:00:40] cos, that would be the failmail... [19:02:28] ouch! [19:02:59] awight: I'm not seeing the second line [19:03:09] oh [19:03:18] now i see it :-) [19:03:34] Jeff_Green: oh, relevant detail--they *don't* consistently fail, each job has a bunch of successes, then a bunch of failures. [19:05:05] i wonder if this could be a pipelining thing [19:06:17] how so [19:06:18] ? [19:06:37] like if php tries to shove 200 messages into the same mail connection [19:06:50] hmm. yeah [19:06:52] and postfix has a limit blow that number [19:07:10] not sure, postfix is mostly using default settings [19:07:15] ejegg do you recommend a good way to keep the db updated/usable on my local? [19:07:17] and I would expect it to scream in the logs [19:07:23] should I just redo dumps from time to time? [19:09:35] awight: we're hashing fr-tech too? [19:09:43] fr-tech+wmf_campaigns-bounce@wikimedia.org [19:09:57] Jeff_Green: first answer here is not that informative, http://stackoverflow.com/questions/1543153/is-there-a-limit-when-using-php-mail-function but maybe we're hitting the socket limit? [19:10:11] Jeff_Green: wait what's the problem with that addy? [19:10:22] that's just to help with email filtering... [19:10:28] (inbox) [19:10:38] it's concievable but I would expect postfix to log any reject [19:10:50] awight: should be fine, just unexpected [19:12:04] drupal always is mailing by /usr/sbin/sendmail and not a connect to port 25, right? [19:12:16] that sorta rules out the pipelining question... [19:13:07] Jeff_Green: well, that SO answer says that php mail() actually connects to port 25, for each letter [19:13:18] with enuf in TCP_TIME_WAIT, we're screwed. [19:13:33] but postfix sees it as the uid of the webserver [19:13:47] which doesn't make sense for a tcp socket [19:13:54] oh [19:13:59] bad SO info?? :D [19:14:09] so it would seem, not sure yet [19:15:30] Jeff_Green: actually, that paragraph was plagarized from an official-looking "note" on the php.net manual page for mail() [19:16:03] me reads src [19:18:11] ok [19:18:29] i've prolly got to turn my attention to a deploy for a bit [19:19:30] sure. ping me when restored table is available! [19:19:43] which table do you need again? [19:19:56] wmf_contribution_extra [19:20:11] fwiw, net.core.somaxconn = 1024 [19:22:33] pizzzacat: sorry, had irc minimized! I think we can dump the fredge tables with a date condition to get just the updates. [19:26:25] ejegg: do u have start/end times for your recalc job? [19:27:21] it started and stopped a few times, but it's all in syslog. Let me get that [19:29:25] ejegg: the TY job was uncharacteristically slow... it looks the time it takes to run increases by a minute per job, until it started to choke [19:29:37] ejegg I see. where exactly does that dump go again? [19:29:51] Great to shake this stuff out now, though. We'll have lots more of this during the peak of FR'ing [19:30:14] pizzzacat: We manually dump it up on the cluster, then copy it local and insert it into our dbs [19:33:34] awight: log times say it ran from 06:35-07:02 and restarted at 16:51, running till 18:43. All UTC, I guess [19:33:53] and 6:35 must have been the logrotate time [19:34:03] ejegg: thx. it is cleared from the list of suspects, in that case [19:34:12] right [19:34:34] we had failures from 14:44 to 15:48 [19:34:41] awight: I don't see any indication in the logs that postfix was unhappy, could we add some logging to drupal in case it happens again? [19:34:57] Jeff_Green: definitely, that "@mail" error suppression is real dirty [19:35:03] ha [19:35:24] Jeff_Green: I did find that the jobs take longer to run each time, leading up to the first failure [19:35:38] weird [19:35:40] So, concerned that this is a load-related issue that we'll hit again [19:36:21] ejegg totally, just meant *where* is our local db? [19:36:31] it's been a while since I did that one dump [19:36:39] and in the config it just says 'fredge' [19:37:38] pizzzacat: I'm running mysql on my dev machine [19:37:55] You probably are too, if you've got a local wiki install and everything [19:38:07] ohhh duh ok [19:38:26] awight: maybe so. but postfix should be able to handle insane amounts of mail on this machine [19:43:21] xmcm cx xxz d d [19:43:29] ... and then she hit the btn [19:43:44] Hi, I wanted to report a typo in the German fundraising banner. I went on meta but I found out the typo is _within_ the %AVERAGE% expression. What should I do? [19:44:48] WikiGnom: you could look here to figure out who was responsible for putting the banner up, https://meta.wikimedia.org/wiki/Special:CentralNoticeLogs?log=bannersettings [19:44:48] (The typo is that there needs to be a space between the amount ("20") and the currency ("€").) [19:45:53] awight: thanks, but I don't understand that log. it's too technical for me. [19:46:51] WikiGnom: I think this guy will know more about the WMDE banners, https://meta.wikimedia.org/wiki/User:Christoph_Fischer_%28WMDE%29 [19:46:59] WikiGnom: thanks for the report! [19:52:07] Jeff_Green: fwiw, php mail() does use shell and sendmail to do its dirty work [19:52:29] https://github.com/php/php-src/blob/master/ext/standard/mail.c#L335 [19:56:32] awight: ok [19:59:36] awight: maybe there's a concurrency limit with that command [20:13:21] atgo: There's a pretty serious-looking new issue, fyi. https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2145 [20:13:36] the image is broken [20:13:43] atgo: slow load? [20:13:49] maybe [20:14:14] what happened here? [20:14:21] like... is this new? [20:14:28] obviously we haven't odne the spike yet :P [20:14:56] atgo: it's possible that some other things were going on, but the teal deer is that we currently aren't able to send thank-yous at normal speeds. [20:15:10] Which will be a huge pain during the peak season. [20:15:19] ha i just had to google that. [20:15:37] atgo: donno, I'm ready to let close this spike for now, but want to do some of the follow-up work. [20:16:19] ok.. i'm going to add it for later in the quarter [20:16:20] mostly, make the TY job more robust in the high-load condition [20:16:23] well. [20:16:24] yeah word [20:16:26] ok [20:16:39] as in, not right in the current sprint, but let's try to get to it before dec [20:16:57] is it failing to send TYs or just slow? [20:19:03] awight ^ [20:21:21] atgo: it's failing. [20:21:41] atgo: we lost 660 letters today, and the way in which it failed actually damaged the contribution records [20:21:46] yeowch [20:22:09] then... I left elephant footprints all over the place and damaged the entire db, trying to fix this small thing :-/ [20:22:25] So, I've updated the spike with my conclusions [20:22:38] I think we need #2141 right away [20:22:58] #2146 is really easy, and gives us some info that might help diagnose the underlying issue in the future [20:23:25] #2147 is also easy, and will give us a more complete solution in any high load situation, but can wait [20:25:08] Jeff_Green: haz restored table? [20:25:20] oh, right [20:25:24] what's the table you need? [20:25:36] so awight what's happening is we aren't sending thank yous to some people? [20:25:44] 2141 fixes it? [20:25:47] Jeff_Green: wmf_contribution_extra [20:25:52] awight: ok [20:26:02] atgo: there's some underlying problem, maybe just high load [20:26:11] awight and Jeff_Green your IRC colors are the same today and it's throwing me off. grumble. [20:26:11] :P [20:26:16] awight: fwiw load never went over 2 on the box [20:26:19] it's causing the TY job to bog down (which should not happen regardless) [20:26:23] ha [20:26:24] over 2? [20:26:38] dang. there's those colors again. [20:26:50] then, we start failing to send emails and screwing up the records by marking them as "failed ty", which prevents us from recovering and resending those [20:27:00] oh dear. this sounds like a rats nest [20:27:06] Jeff_Green: not sure it was CPU load [20:27:19] hey ejegg i'm pinging you in the other channel [20:27:22] atgo: I think I have a solid plan at the end of that spike [20:27:34] awight: ya, just saying, there doesn't seem to be a system load correlation anyway [20:28:29] ok awight... is this urgent enough in your opinion to disrupt the "plan"? [20:28:48] or can we add it as top priority for the next sprint? [20:29:08] because you have "should have" as the priority ahorita [20:29:21] I think I set the followup tasks to must have [20:29:42] 2141 [20:29:45] awight: did anything change that you can think of other than the switch to VERP addresses? [20:29:57] I think... we are screwed if we go into the winter FR with this issue. But we can live for now [20:30:24] ejegg: you mind if I turn off TY activity recording for a minute, to see if it's affecting performance [20:30:27] ? [20:30:30] come to think of it,could it be as silly as the VERP address being ugly from a shell command perspective? like with unescaped special characters? [20:30:41] Jeff_Green: hehe that would be a gas [20:30:54] I think the To is sent over stdin though [20:31:05] yeah [20:31:16] the to is, but not the envelope sender address [20:31:32] both envelope address have to be on the command line iirc [20:31:38] awight: go for it [20:31:56] VERP should just be alphanumeric, with some dots [20:31:58] err I meant the envelope recipient [20:32:00] ok [20:32:13]