[17:47:34] Jeff_Green: *waves* [17:58:04] * Jeff_Green mwalker|away: *waves back* [17:58:08] what's happening [17:58:31] on friday I managed to get phpmailer into the git repo with full history [17:58:35] so now that's ready to go [17:58:56] cool. i'm working on this as we speak actually [17:59:01] awesome! :) [17:59:17] i created /srv/tools/ for these--should I move twig there too? [18:02:26] sounds reasonable [18:02:32] ok [19:04:33] mwalker: I think all the listener stuff is in place now [19:05:14] awesome -- give me just a sec -- I'm in the middle of poking adp to schedule time off [19:05:22] oh the horror [19:08:45] ok [19:08:52] this system sucks harder than donation interface [19:08:54] but! done [19:09:26] hooray [19:09:27] so; first steps -- where is syslog and apache error going? [19:09:36] I have some test messages I can have it process [19:09:40] sec [19:10:04] i thought I'd already dealt with it but apparently not [19:10:07] fixing... [19:15:26] if it helps SmashPig log lines will always start with SmashPig [19:15:48] for now I'm just matching on hostname and putting it all in /var/log/remote/payments-listener-misc [19:16:08] we can break it out as needed. fixing firewall rules now to allow logging at all... [19:16:16] kk [19:16:34] ok, syslog goes to indium:/var/log/remote/payments-listener-misc [19:16:53] checking apache error log, i think we foisted that on syslog too [19:17:42] yup, all to syslog [19:17:49] fixing loudon... [19:24:11] yay! so I see the logfile on loudon! [19:24:25] next -- what is the URL I need to use to hit smashpigs http listener? [19:25:08] https://payments-listener.wikimedia.org/smashpig/ ? [19:25:30] I forget. looking... [19:27:17] oh right, I halted apache last week. [19:29:32] apparently we're running out to get food quickly [19:29:42] k. [19:29:56] but! to test that it's working on your end https://payments-listener.wikimedia.org/smashpig/adyen/listener should through a whole bunch of crap into syslog [19:30:01] and give you a SOAP error [19:30:18] ok. i've got apache up and I'm just digging up the URL [19:31:02] i don't see ./adyen as a folder, should that be there? [19:42:20] mwalker: ok I think it's sorta working now, but the URL is .../SmashPig/... [19:48:19] * marktraceur is confused by the GET_ORDERSTATUS return values, apparently they're significantly different from INSERT_ORDERWITHPAYMENT returns [19:54:28] Jeff_Green: yay! I get the soap error (and in case you didn't figure it out -- adyen is a virtual folder) [19:54:37] but! I don't get and shtuff in syslog [19:54:43] yeah, that's what I was wondering re. adyen [19:55:04] hrm re. shtuff [19:55:20] lemee poke in the outgoing SSL IPs and we'll take a look at logging [19:58:41] mwalker: will outbound always be https? [19:59:06] i hope so! [19:59:18] at the moment thats the case [19:59:27] ha. ok i will only open outbound tcp/443 then [20:01:12] ah -- also mail! [20:01:27] yup. we'll have to test that [20:01:50] i think it should work now but it relays via mchenry and I'm not sure we'll like what happens with headers [20:06:53] ok let's get logging working. what's the URL you're using to test? [20:10:31] https://payments-listener.wikimedia.org/SmashPig/adyen/listener [20:11:41] should be a couple of 'info' and 'warning' and possibly 'debug' level statements [20:17:53] i see those going in /var/log/syslog locally [20:19:09] so; just not making it to loudon? [20:19:17] I am tail -f /var/log/remote/payments-listener-misc [20:19:22] and getting nothing [20:20:34] yeah, that's odd. looking [20:21:34] I'm seeing it on indium [20:22:10] oh, looks like firewall rules are still blocking syslog to loudon. fixing [20:25:48] yay! fixed. [20:26:04] awesome [20:26:09] ok -- SmashPig configuration [20:26:27] ya. i threw that config file from your homedir on al, what needs tweaking? [20:26:54] default/logging/failmail/inst-args [20:27:00] should probably point to fr-tech@wikimedia.org [20:27:25] 'inst-args' => array'fr-tech@wikimedia.org', [20:27:28] gah [20:27:35] 'inst-args' => 'fr-tech@wikimedia.org', [20:28:26] default/data-store/stomp/lib-path needs to point to the place where you pulled down Stomp [20:28:45] /default/include-paths needs to have the location of phpmailer [20:30:27] and /default/data-store/stomp/queues/limbo lets have point to '/queue/sp-limbo' [20:30:39] also add 'verified' => '/queue/sp-donations' [20:32:14] phpmailer--is that just the top level path of the phpmailer project? [20:32:42] i.e. /srv/tools/phpmailer ? [20:33:00] yep [20:33:26] k [20:36:06] cc-limbo becomes sp-limbo? [20:36:38] ya -- for the moment [20:36:49] I just want to make sure we dont bork existing queues whilst we're testing [20:36:57] k [20:37:00] blasting that config out [20:37:59] interestring new logbarf [20:38:48] ah [20:38:55] shoot -- that's probably the failmail addresses [20:39:05] should I turn it back into an array? [20:39:10] yep [20:40:10] more funbarfloggy [20:41:19] ooh [20:41:57] what URL were you using? [20:42:26] https://payments-listener.wikimedia.org/SmashPig/PublicHttp/adyen/listener [20:43:01] oh; you shouldn't need the publichttp bit [20:43:03] https://payments-listener.wikimedia.org/SmashPig/adyen/listener [20:43:05] works just peachy [20:43:27] because SmashPig does the rewrite to publichttp/public_http_listener.php [20:43:36] k [20:43:45] does logging look good now? [20:43:48] yeppers [20:44:03] next task! there's a cron job that we'll need to run [20:44:26] I cant recall if I asked you about running that via jenkins [20:46:12] you haven't mentioned yet afaik unless it was when I was drunk [20:46:21] or undercaffeinated [20:46:47] ok! I know it was all very confused when we were putting together things -- so its probably on me [20:46:57] in any case -- there's a PHP script that needs to run pretty routinely [20:47:05] any reason cron is not adequate? [20:47:35] if we can setup a slave jenkins setup - then we have easier control over it's execution / get failmail from it [20:47:46] and turn it on/off when we do upgrades [20:48:07] simple cron will work [20:48:13] but I'm always greedy [20:48:48] let's go cron, because that can be puppetized and we don't have the whole jenkins is a single point of likely failure [20:49:17] that spoken by the person who implemented the cron based jenkins watcher script to lift jenkins up when it faceplants [20:49:35] hehe [20:50:23] ok -- we're running the SmashPig/Maintenance/StompJobRunning.php [20:50:29] *StompJobRunner.php [20:50:44] ok. how frequently and/or what time? [20:51:18] full up -- probably every 5 minutes [20:51:20] or more [20:51:36] k. run it as the same user as the webserver? [20:51:40] yep [20:51:45] k.o. [20:53:39] uh -- it apparently needs some options [20:53:40] php StompJobRunner.php --config-file ../config.php --config-node adyen [20:53:45] ok [21:00:39] ah -- cool [21:00:44] got some bugs I need to fix [21:00:51] ok. should I stop the cron? [21:00:54] yep [21:01:15] it'll just generate cronspam [21:01:29] this is fun though: [21:01:30] Could not send email to fr-tech@wikimedia.org. PHP Mailer had exception. | | Invalid address: smashpig-failmail@thulium#012#0 /srv/www/org/wikimedia/listeners/SmashPig/Core/MailHandler.php(119): PHPMailer->SetFrom('smashpig-failma...')#012#1 /srv/www/org/wikimedia/listeners/SmashPig/Core/Logging/FailmailLogStream.php(131): SmashPig\Core\MailHandler::sendEmail('fr-tech@wikimed...', 'FAILMAIL -- ERR...', 'A problem has d...', 'smashpig- [21:02:47] hah [21:02:52] yay phpmailer [21:04:22] oh! that's what the second inst-arg was to Failmail [21:04:44] so default/logging/log-streams/failmail/inst-args [21:04:55] needs to be array( , ) [21:05:11] * mwalker thinks I need a better system for this [21:06:05] default/logging/log-streams/failmail/inst-args => array('fr-tech@wikimedia.org', 'smashpig-failmail@thulium.wikimedia.org'), [21:06:24] * Jeff_Green fighting puppet variables [21:14:20] mwalker: ok correction made and cron restarted [21:14:34] oh -- that wasn't a cron error [21:14:39] the cron error is actually a bug [21:14:48] oic. should I stop that job? [21:14:52] yes please :) [21:16:28] but -- cool; the ball is now back in my court to fix the bugs I'm finding [21:16:46] can I deploy software fixes? or do I still need your intervention for that? [21:18:36] you can do it [21:18:47] off of boron? [21:18:50] payments_code_update and payments_rsync_blaster ALL:ALL [21:18:50] yep [21:18:54] kk [21:27:11] mwalker: are you able to tell whether or not this rig is talking to activemq properly now? [21:27:43] uh -- not yet [21:27:48] k [21:27:55] quick thing we can do [21:28:12] if you run StompJobRunner.php from the Maintenance directory directly [21:28:15] it should run [21:28:55] not do anything -- but it should at least vomit if it cannot talk to stomp [21:29:46] so: [21:29:56] /srv/www/org/wikimedia/listeners/SmashPig/Maintenance# php StompJobRunner.php [21:30:00] or does it need args? [21:30:03] ya [21:30:18] so --config-file ../config.php --config-node adyen [21:31:08] lots of fail trying to connect to localhost:61613 [21:31:51] ah [21:31:54] hang on [21:32:14] ok [21:32:35] /default/data-store/stop/uri => 'tcp://silicon:61613' [21:32:45] */default/data-store/stomp/uri [21:33:34] which file? [21:33:38] config.php [21:34:49] ok that looks sorta better. fixing firewall now [21:43:26] had to get leslie involved, appears to be blocked at hardware firewall [21:45:51] whoo [21:46:08] server kitties! [21:47:06] Attempting to STOMP subscribe to '/queue/job-requests' on 'tcp://silicon.frack.eqiad.wmnet:61613' [21:47:10] Pulling new object from queue [21:47:14] STOMP did not return an object [21:47:22] that's good right? [21:48:49] yeppers [21:48:56] there's nothing in the queue [21:49:00] but it can talk to it [21:49:06] so as expected [21:49:07] glorious [21:49:22] are we at a point where I can stop for the day? [21:49:35] i.e. without leaving something insecure/destructive hanging? [21:50:34] yes [21:51:19] ok