The above diagram is a quick look at a very high level conceptual view of how one might go about structuring their web applications to take advantage of Amazon’s SNS and SQS.
Amazon SNS - http://aws.amazon.com/sns/
Amazon SNS provides a Simple Notification Service for your apps. A line of thinking that maybe helpful would be viewing your app as sending a text message to the cloud. I use cloud as a generic term, because you are actually pushing your message to an unknown which will then re-route it to the system or services you wish to inform. Re-route examples being; email, an app, or SQS. The key here is that you’re not doing any P2P messaging. You’re invoking a central bus that figures out where your message should go. If it still hasn’t hit you yet, it means you don’t have to change your app every time an endpoint changes.
Amazon SQS - http://aws.amazon.com/sqs/
Amazon SQS is a Simple Queue Service for your apps. This can be thought of as a stack of papers on your desk. Each paper contains a task / message of what you need to get done. The stack of papers sits neatly ordered waiting for you until you come and pull off the next piece of paper. In the above diagram our queue is holding a buffer full of tasks that our apps need to react to… Or perhaps just relevant information that our apps need to know. E.g. something happened here…
The flow of the diagram is as follows:
1. App 1 deletes a User from it’s database.
It just so happens that App 2 and App 3 hold references to that User, but they don’t share the same database. So, App 1 needs to send a message to App 2 and App 3 that something happened (i.e. User_55_Was_Deleted).
2. App 1 sends a notification that User X was removed.
We could have just sent a message from App 1 directly to App 2 and then to App 3. However, what happens if we decide App 3 needs to be taken down. Well, our code should account for this with various error catching schemes, however, we also don’t want App 1 to be waiting for a timeout for the rest of it’s existence simply because App 3 was taken offline. Async or not, it’s still CPU cycles and network wasted.
3. Amazon SNS receives the message and processes it.
Amazon will take the message it just received from App 1 and look at it’s processing rules to figure out what to do with it next. Should we email someone? Should we send a note to another app? Should we invoke a url? Etc.
4. Amazon SNS sends the message to App 2 and App 3 SQS.
It may make more sense to ‘KISS’ (avoid more moving parts), but the reality of the situation is… What happens when App 1 sends a notification and App 2 is down? Amazon SNS is NOT going to hold on to your message. The point of SNS is real-time quick small messages. If your app is down, SNS will not keep spamming notifications until it comes back up, at least it should not. SNS is not a queue. We need to use SQS to hold on to our message until our apps are back online and ready. Scenario #2 is; perhaps our apps shouldn’t be processing certain messages in the middle of the day if not necessary. Maybe it’s better to wait until the wee hours of the morning when no one is on the system to run a potentially processor intensive task.
5. App 2 routinely checks Amazon SQS for messages to process.
App 2 can run a cron job to check for new messages and either process them now or leave them in the queue. App 2 and App 3 simply pull down the message. Note, they have their own feeders (SQS), so one doesn’t pull a note down from the others stack.
Summary:
The big take away from this is to realize that app to app communication is faulty and dangerous. Instead of rolling your own Service BUS, consider using something that is already setup and managed and distributed for you. And in the end, your apps will be more modular and everyone will be happy. Yay for event based programming.