SCATA Logo

SCATA mailing challenges

Posted By Grant Forrest |  last update 17-Sep 2010

One of the tasks given to me as webmaster was to implement a web-based mailing solution that allows a committee member to send a membership-wide email.

Not that much of a challenge, you are probably saying, just use PHPMailer or MailChimp or something like that. Mail Chimp would have been the easy way out but I did dip into PHPMailer for the final step of actually sending an email.

The SCATA membership list is some 400+ members in size, but the server's limit is 100 emails per account per hour, so I couldn't just bang them all out in one go.

The trick is to use an execution pause in the mail script. Something like :

global $mail_interval;
while ($member = mysql_fetch_array($res)) {
  $mail_res = phpMailer(
    $member["email"],
    $recip_name,
    $subject,
    $msg,
    $format,
    $distribution,
    $sender,
    $sender_name);
  // save the mail result
  CreateMailResult(
    $campaign_id,
    $sender,$member["email"],
    $recip_name,
    $mail_res);
  // sleep for $mail_interval
  // will not work with Windows servers pre PHP5 / All servers PHP4
  usleep($mail_interval * 1000 * 1000);
  // after the pause, we have to reset the max execution time
  set_time_limit($mail_interval * 2);}

This works fine, but the next problem is how to feed back progress to the browser. I experimented with various methods and one by one rejected them all as unsatisfactory. At the core of the problem is that scripts with sleep() or usleep() will have their output cached. It doesn't matter what tricks you try with ob_start() etc, you just won't be able to feed back progress to the browser. Even using AJAX, you have the same underlying problem.

What I really wanted was a way to run the script in the background and allow the user to get an update as and when required. Each mail transaction is logged in a MySQL table, so it's simple to fetch this information on demand.

I thought about running the mail script in the background using exec() but for security reasons, exec() is disabled on our server.

After a lengthy trawl of the coding forums, I hit on an idea. Why not open a TCP socket connection, run the script and then just chuck away the result ? For example :

  // Create a socket to the mail script page and then return immediately
  $fp = fsockopen($host, 80, $errno, $errstr, 30);
  if (!$fp) {
    $err = "$errstr ($errno)
\n"; } else { $out = "GET $file HTTP/1.1\r\n"; $out .= "Host: $host\r\n"; $out .= "Connection: Close\r\n\r\n"; fwrite($fp, $out); fclose($fp); }

Once the mail script is running, we can output a simple page saying "Mail Campaign in Progress" and the socket connection has launched our mail script in the background.

Not perfect, but reasonably clean. A few snags still to tackle though.

If the script hits an error, it will just die, so I probably need to get it to write errors to a file rather than try to return them to the browser which has thrown the connection away.

Also, if you make a mistake and click "send" accidentally, there is no simple way of stopping the mail script, short of restarting Apache. I guess in theory you could hunt the httpd process ID down and try to kill it off, but how would you match the PID to the name of the script ?

Maybe next time I'll just try Mail Chimp ;-)

13 comment(s)


Posted by paul on 25-May 2010 09:05

ps -A ps -A | grep whatever :-)

Posted by Grant on 25-May 2010 09:05

Yeah but what is the "whatever" ?!

Posted by paul on 25-May 2010 17:05

ps -A returns a list of all the running processes. To filter the results pipe the list to grep ps -A | grep apache will give give you the details of apache . it will also take wildcards ps -A | grep ss* man ps

Posted by Grant Forrest on 25-May 2010 17:05

On my system apache ps -A just produces lots of "httpd" to indicate the various apache worker threads, but how do I find out what page that particular thread is processing ? BTW I've worked out the error_reporting() part : $old_error_handler = set_error_handler("MyErrorHandler"); which writes errors to a file.

Posted by paul cooper on 28-May 2010 15:05

what does ps -A grep apache show ? and whats lsof | grep apache

Posted by Grant on 22-Aug 2010 13:08

If I do : ps -A -F user 32465 32358 0 25709 9356 2 13:14 ? 00:00:00 /usr/bin/php /home/user/public_html/secure/admin/campaign_progress.php which is the running mail script. Sometimes it just seems to die though :(

Posted by Grant on 17-Sep 2010 18:09

Well, so much for my brilliant idea of running the mail script as a background httpd process. Every time Apache does a graceful restart (which on our server is about every hour due to log rotation), the script dies. Not good ! It'd taken me ages to find an alternative but I'm testing a new method that looks hopeful.
I've replaced my http connection with a call to

shell_exec()

Looks something like this :

$output = shell_exec("php-cli -f
'/home/sitename/public_html/campaign_progress.php' $campaign_id $sid >
/home/sitename/public_html/mail_debug_log &");

Many issues came up with this approach. shell_exec() is disabled for security reasons on a lot of systems. I had to re-enable it on our server. Next, the PHP command-line interface has a poorly documented bug (or is a design flaw?). http://bugs.php.net/bug.php?id=11430

Essentially what was happening is that a web script running as PHP-CGI was calling a command-line script. I didn't realise that you had to make it explicit that you wanted your shell script to run as CLI - note the use of "php-cli" rather than just invoking PHP using "php". If you don't do this, you get an infinite loop where the PHP-CGI binary effectively calls itself. This crashed my web server many times before I realised what was going on. Fortunately it was 1AM and nobody was about to feel the effects of my mistakes. I had to change my background mail script to begin :

#!/usr/bin/php-cli -q

because it was now running as a PHP-CLI process instead of an Apache (httpd) process.
Testing is underway but it's looking like this can stand up to a graceful Apache restart.
For some reason the script runs OK with 644 file permissions. I assumed it would need to have executable permissions but apparently not.
I also found out how to monitor the process from the command line :

ps -AF | grep account_username

That will show you detailed info on all the processes running under that user.
I'm looking forward to some trouble-free mass-mailing.


Posted by Paul Gardner on 23-Sep 2010 17:09

You could store the e-mail in a MySQL table, and have the actual e-mailing done via a script ran by cron? This would then give you the ability to queue multiple e-mail campaigns. The cronjob could update the status as it's running and store that status in a MySQL column. This would effectively detach the mailing process from httpd, getting around the httpd restart issue. Feedback to the user could be done via a simple script which reads the status column from the table, and use a meta tag to refresh the page every x seconds/minutes.

Posted by Grant on 24-Sep 2010 09:09

Hi Paul The script logs each mail transaction to a MySQL table so campaign progress can be reviewed as you go along. I considered using a 40-second cron job but couldn't work out how to loop through the mailing list with a static php command.

Posted by Paul on 27-Sep 2010 12:09

Maybe something like this?

// Pull list of members who subscribe to email newsletters

$stmt=$database->query("SELECT me_email FROM members WHERE me_subscribed=1");

$members=$stmt->fetchAll(PDO::FETCH_OBJ);

 

// Pull all unsent campaigns

$campaigns=$database->query("SELECT ca_id,ca_subject,ca_content FROM campaigns WHERE ca_status=0");

$campaigns=$stmt->fetchAll(PDO::FETCH_OBJ);

foreach($campaigns as $campaign) {

// Mark campaign as 'in progress'

$stmt=$database->prepare("UPDATE campaigns SET ca_status=1 WHERE ca_id=:caid");

$stmt->bindParam(':caid',$campaign->ca_id,PDO::PARAM_INT);

$stmt->execute();

foreach($members as $member) {

$mail=new Mail();

$mail->setFrom('[email protected]');

$mail->setSubject($campaign->ca_subject);

$mail->setContent($campaign->ca_content);

$mail->setRecipient($member->me_email);

$mail->send();

// Sleep if needed here

// Update progress here too

}

}

 


Posted by Grant on 28-Sep 2010 07:09

Would that not re-send to the entire list every time the Cron job is run ?


Posted by Paul on 28-Sep 2010 14:09

No - the second query which pulls the campaign data will only pull campaigns which haven't started (ca_status of zero).  As soon as the campaign beings processing, it updates the ca_status to 1 which means the next time the cron runs, it won't deal with that campaign.  Although you'd probably want to have a lock column to prevent multiple crons running if you're throttled on how much mail you can send.

That's just a foundation, there's lots more you can do once you get to that stage.  I'd consider using a m2m table to keep track of who has been mailed in the current campaign.  That would make dealing with errors trivial and the solution would also then cope with server restarts, intended or otherwise.

For a cancel function, you could just do a UPDATE campaigns SET ca_cancelled=1 WHERE ca_id= and have the cron check the cancel flag is still zero before calling $mail->send().


Posted by Grant on 21-Jan 2016 22:01

Update 14-Jan 2016

Have realised that PHP 5.6 no longer ships with php-cli binary executable. Happy to report that :

#!/usr/bin/php

and

shell_exec()

still work :)


show all blogs