Sending emails using Laravel queues and Beanstalkd

Simple example demonstration sending emails in laravel with queues and step by step guide from installing and configuring beanstalk, supervisor and Laravel Queue.

Steps to follow

Configure the Laravel Environment
Installing Beanstalkd and Supervisor
Creating Artisan Commands to process the emails
Create a Laravel Job to procces the Queue
Setting up Cron to automate email sending

Configure the Laravel Environment

If you are familiar with configuring Laravel .env file and config files then this will step will be easy for you. All our configurations related to this tutorial goes either in the specific configuration files or .env file. If you have any difficulty in configuring files in your projects, the official documentation gives incredible data, and there are a couple of examples related to each service which we can use in Laravel.

Laravel has a clean, simple API over the popular SwiftMailer library and provides drivers for SMTP, SendGrid, Mailgun, Mandrill, SparkPost, Amazon SES, PHP's mail function, and sendmail, allowing you to quickly get started sending mail through a local or cloud based service of your choice.

In this tutorial we will be using Amazon SES to send emails. You can choose any email sending service like Mandrill, SendGrid, Mailgun and more are available now. We have a detailed tutorial on CONFIGURING LARAVEL TO USE YOUR AMAZON SES CREDENTIALS will be helpful for this step.

Installing Beanstalkd and Supervisor

Beanstalk is a simple, fast work queue.

"Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously." - Official Website

We will be sending bulk emails from Laravel to demonstrate how to use queues. While sending email, the application put the job to send emails in a queue, and our workers will take each job and process synchronously.

Installing Beanstalk on Ubuntu

sudo apt-get install beanstalkd

Public access

We need to enable public access in order to access the beanstalkd queue from host PC.

This is only for development, because we will be using beanstalk console on the host PC to watch the queue of Beanstalkd running in a virtual machine.

Don't do this on production server.

Go to sudo nano /etc/default/beanstalkd and change BEANSTALKD_LISTEN_ADDR=127.0.0.1 to BEANSTALKD_LISTEN_ADDR=0.0.0.0 for public access.

Now restart the beanstalkd service:

sudo service beanstalkd restart

Now, let's install beanstalk console.

Beanstalk console installation

Admin console for Beanstalk queue server

If you want to see the queue while developing the application this console is very handy. You can see the number of jobs and inspect every job or delete it. Click the link above to see more details.

Run this command somewhere on your PC where you have PHP 5+ and Composer installed:

composer create-project ptrofimov/beanstalk_console -s dev

It will create a folder for beanstalk console. Navigate to that folder and start php server with this command:

php -S localhost:7654 -t public

Now if you check http://localhost:7654 in your browser you should get a screen like this:

beanstalkd console first run

Add remote server to check beanstalkd queues. Leave the port to default and change the server name. You can use the IP address or the server name where you have install beanstalkd.

Configure Laravel

You have to require pda/pheanstalk in your composer.json file.

In your application root folder type:

composer require pda/pheanstalk

and when asked for version type: 2.1.*.

Or you can just add this line to your composer.json require section:

"pda/pheanstalk": "2.1.*"

and run composer update.

When composer is done updating you have to change the Default queue driver in app/config/queue.php file to beanstalkd.

'default' => 'beanstalkd',

Installing Supervisor on Ubuntu

Supervisor is a process control system. Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems and will automatically restart your queue:work process if it fails. To install Supervisor on Ubuntu, you may use the following command:

sudo apt-get install supervisor

Configuring Supervisor

Supervisor configuration files are typically stored in the /etc/supervisor/conf.d directory. Within this directory, you may create any number of configuration files that instruct supervisor how your processes should be monitored. For example, let's create a laravel-worker.conf file that starts and monitors a queue:work process:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work 
directory=/var/www/laravel-queue-demo
autostart=true
autorestart=true
user=vannstudios
numprocs=10
redirect_stderr=true
stdout_logfile=/var/www/laravel-queue-demo/storage/logs/supervisor.log

In our tutorial, the numprocs directive will instruct Supervisor to run 10 queue:work processes and monitor all of them, automatically restarting them if they fail.

Starting Supervisor

Once the configuration file has been created, you may update the Supervisor configuration and start the processes using the following commands:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-worker

# If you have multiple configuration file. you can restart all your worker together using the following command

sudo supervisorctl restart all

Creating Artisan Commands to Process the Emails

While creating console command our intension is to add the emails to queue as well as adding the Command to the Laravel task scheduler.

php artisan make:command SendEmails

Now we will add our logic in the handle function to send emails to the Job.

public function handle() 
{
	$subscribers = Subscriber::where('status',1)->get();
	$chunks = $subscribers->chunk(100);
        foreach ($chunks as $chunk) {
	    ProcessEmail::dispatch($chunk->toArray())->onQueue('aws');
	}
}

So here we have made our subscribers list as a Chunk. The chunk method breaks the collection into multiple, smaller collections of a given size and will reduce the time taken for a single queue to process the emails as will as memory in case if we are adding more data in the email content or template.

Create a Laravel Job to Process the Queue

Now as we have seen in our step 4 we have added all our emails to the Job ProcessEmail, but we havn't created it. So now we will go ahead and create one.

php artisan make:job ProcessEmail

So now we have our Laravel Job to send emails. You can use any email service provider like Sendgrid, Mailgun, Amazon SES to send emails. Each vendors have their own API and examples showing how to send emails in php, Laravel and other frameworks. If you have already gone through our tutorial on Sending bulk email in Laravel with Amazon (AWS) SES, then you can skip this step, else going through the tutorial will be helpful to do this step. In our example we have used a Controller, so here we need to change it to our ProcessEmail Job, Don't forget to pass the email ids to the recipients list.

Setting Up Cron to Automate Email Sending

Our last step in this example is to automate the email sending, for that we will add our Artisan Command in the Kernal.

protected $commands = [
      \App\Console\Commands\SendEmails::class,
  ];


<!--To schedule the emails daily at 10:00, we will add the following code in the schedule method. -->


protected function schedule(Schedule $schedule) {
    $schedule->command('command:send-email')->dailyAt('10:00');
}

Now we have automated our email sending to our subscribers using Laravel queues and Beanstalk.

Hope its helps you. Good luck. Cheers!

Recent resources