How AWS EFS works (Best Tutorial 2019)

 

AWS EFS

How AWS EFS works (Complete Tutorial 2019)

This tutorial explains the complete process of how AWS  Elastic File System (EFS) work with best examples. And also explain how to configure EFS with Elasticsearch in 2019.

 

The Elastic File System (EFS) provides a option for storing information on Amazon Web Services (AWS).  This tutorial also configure EFS with EC2 setup. One of the services that EFS supports is Elasticsearch.

 

The Elasticsearch is an open source search and analytics engine. You use it to perform various sorts of analysis, such as checking out your log files or performing real-time application monitoring.

 

When you combine EFS with Elasticsearch, you gain considerable functionality in turning your EC2 server into something that can help your organization control precisely how people use and interact with your server.

 

You can also begin avoiding those embarrassing slowdowns or complete application failures that cause people to go to any other site they can find.

 

Elastic File System (EFS) Features

Elastic File System (EFS)

 

Introducing EFS

How AWS EFS works

Most operating systems today provide about the same paradigm for managing files. A drive contains a root directory. This root directory can contain as many folders as needed to organize the data that the drive contains. Subfolders contain content that becomes more specific as the hierarchy depth increases.

 

Each folder or subfolder can contain files that contain data. Depending on the characteristics of the file system, the files can contain simple, complex, fully populated, or sparse data. Some files can even contain multiple data streams. EFS replicates the most common features of file systems found on operating systems today.

 

It doesn’t replicate every feature, but it replicates enough features to make using the file system easy. An EFS drive looks like any other drive you use on any other operating system.

 

You can use EFS independently, in conjunction with other AWS services, or through a gateway (to make it appear as part of your local network). The most common service to connect to EFS is Elastic Compute Cloud (EC2), covered in another blog.

 

In fact, a major attraction of using EFS is that you can connect multiple EC2 instances to a single EFS drive. This means that all the EC2 instances share data and enable all of them to interact with it in meaningful ways.

 

The common storage uses for EFS are

common storage

  • Application data
  • Big data
  • Analytics
  • Media processing workflows
  • Content management
  • Web application data and content
  • User home directories

 

EFS emphasizes shared file storage, which implies access from multiple services or applications from multiple locations. You need to consider the construction of service before you put it into use — that is, you should develop an understanding of the goals that engineers had in designing it.

 

Because EFS emphasizes shared access, it doesn’t necessarily provide speed or some types of flexibility. The goal is to ensure that services, applications, and devices of all sorts can gain access to specific kinds of data, rather than provide flexible configuration options. 

 

As with other Amazon storage options, EFS scales to meet your current storage demands. Amazon provides the same levels of redundancy and automatic backup for EFS that it does for the other storage options.

 

In addition, Amazon stresses that you pay only for the resources that you actually use to perform tasks. You don’t pay a minimal fee for storage or for setup costs. EFS also provides the same sort of permission system used by the other storage systems.

 

Understanding Network File System version 4 (NFSv4)

NFS

EFS relies on a standardized Internet Engineering Task Force (IETF) file system, NFSv4.

 

Compliance with a standard is important because most Network Attached Storage (NAS) and many operating system vendors also comply with NFSv4, enabling these data sources to interoperate seamlessly and appear to have all the data sources residing on the local machine (even when they’re miles apart).

 

NFS is a mature file system that has seen a use for at least the last 20 years in its various incarnations.

 

Amazon doesn’t provide full NFSv4 support. The page at http://docs.aws.amazon.com/efs/describes the NFSv4 features that EFS doesn’t support, and some of the omissions are extremely important.

 

For example, EFS doesn’t support NFSv4 Access Control Lists (ACLs), Kerberos-based security, lock upgrades or downgrades, or deny share (which means that every time you share a file, you must share it with everyone).

 

These omissions will affect your security planning if you aren’t aware of them and take measures to overcome the potential security problems they present. How much these omissions affect security depends on the kinds of information you attempt to store on EFS, so you must look at your data needs carefully.

 

EFS doesn’t support certain file attributes. It shouldn’t surprise you that EFS lacks support for block devices because Elastic Block Storage (EBS) meets this need. Many of the attribute omissions are for optional features, however, so they may not present much of a problem unless you have specific needs.

 

Be sure to check the Amazon documentation carefully to ensure that omissions, such as namespace support, won’t cause problems for your particular applications.

 

Comparing EFS to S3, Standard – IA, and Glacier

Comparing EFS

When comparing EFS to S3, S3 Standard – Infrequent Access (Standard – IA), and Glacier, the first thing that comes to mind is money. EFS is the most expensive of the Amazon storage options, and S3 is the least expensive.

 

The difference that you pay between the various storage options is substantial, so you need to consider how you use storage in connection with your business carefully. Using all these services together often obtains the best value for your storage needs.

 

For example, if you really don’t plan to use some of your data for an extended time (with extended being defined by your business model), storing it on EFS will waste money. Using Glacier to store data that you use relatively often, however, will most definitely waste time.

 

Considering the trade-offs between the various storage options is the smart way to do things. The additional money you spend on EFS does purchase considerably added functionality.

 

The most important feature that comes with EFS when compared to the other options in this section is the capability to secure files individually. EFS provides full locking support, so you can use it for database management applications or other needs for which locking all or part of a file is essential.

 

Even though S3 and its associated storage options provide support for the group and individual user permissions, you also find that EFS provides better security support.

 

Speed is also a consideration. S3 comes with HTTP overhead that robs it of some speed. However, the main speed difference comes from EFS’s use of Solid-State Drives (SSDs) that make access considerably faster.

 

From a visualization perspective, think of EFS as a kind of NAS, while S3 is most definitely a kind of Binary Large Object (BLOB) Internet storage. Amazon has also optimized S3 for write-once, read-many access, which means that writing incurs a speed penalty that you don’t see when working with EFS.

 

S3 does offer some functionality that’s not possible with EFS. For example, you can use S3 to offer files as a static website — something that you’d need to configure on EFS by hosting it on a web server.

 

The point is that EFS is more like the file system you’re used to using, and S3 is more akin to a special-purpose, blog-based database that provides direct web access at the cost of speed.

 

You can get past some of the S3 limitations by using an S3 File System (S3FS) solution such as s3fs-fuse (https://github.com/s3fs-fuse) or S3FS Node Packager Manager, S3FS NPM.

 

However, even though these alternatives overcome some of the interface issues with S3, they still can’t overcome the security and individual object size issues. (A single object can contain 5GB of data.) 

 

Comparing EFS to EBS

EFS_EBS

The first thing you notice when working with EBS is that it’s the only storage option without a console. You must configure EBS through an EC2 setup. That’s because EBS provides services to only a single EC2 instance. If you need multiple-instance support, EBS won’t work for you.

 

EFS is also designed to support organizations that require large distributed file systems of the type provided by Isilon and Gluster (https://www.gluster.org/).

 

However, in addition to getting a large distributed file system, you can make this installation available across regions using EFS, which is something you can’t do with these off-the-shelf solutions.

 

What you get is a large distributed file system that provides­ region-specific support without your having to build this support yourself­. Because EFS scales automatically, you don’t need to worry about the number of resources that each region requires to satisfy its needs.

 

EBS is restricted to a single region because of its single EC2 instance focus and the fact that it acts like a Storage Area Network (SAN), which provides dedicated network support.

 

Some applications require block storage devices, such as that provided by EBS, to avoid the rules that a file system imposes and gain a speed benefit. For example, Oracle Automatic Storage Management, or ASM, falls into this category.

 

EFS doesn’t answer the needs of a block storage device application; you need a product such as EBS in this case. Oracle ASM still has a file system, but the file system is built into the product, so having another file system would prove redundant and counterproductive because the two file systems would fight each other.

 

Working with EFS

EFS_work

As with the other services, EFS comes with its own management console that helps you create, delete, configure, and monitor the storage that you need. The process for creating and using an EFS drive is similar to the process used for a local hard drive.

 

But some of the automation found on your local system isn’t present with EFS. For example, you must mount the file system (a task that a local setup normally performs automatically for you).

 

As with most local drives, security on an EFS drive relies on both group and individual use access rights. Unlike your local drive, however, an EFS drive can automatically scale to accommodate the needs of your applications (so that you don’t run out of storage space).

 

The following sections describe how to work with EFS drives and ensure that the data they contain remains safe.

 

Starting the Elastic File System Management Console

Management Console

 

1.Sign into AWS using your administrator account.

2.Navigate to the Elastic File System Management Console at https:// console.aws.amazon.com/efs.

3. Click Create File System.

File System

You see the Create File System page, the first step to create a file system is to decide how to access it. You can assign your EFS to one or more Virtual Private Clouds (VPCs).

 

4. Choose one or more VPCs with which to interact. If you have only one VPC, Amazon selects it for you automatically.

5. Scroll down to the next section of the page.

Amazon asks you to choose mount targets for your EFS setup. Amount target determines the locations that can mount (make accessible) the drive.

 

Using all the availability zones in a particular region won’t cost you additional money when you’re working with the free tier. However, using multiple regions will add a charge. Remember that you get to use only one region when working at the free tier.

 

6. Select the availability zones that you want to use and then click Next Step.

You see the Configure Optional Settings page. These settings help you configure your EFS setup to provide special support, but you don’t have to change the settings unless you want to. This page contains two optional settings:

 

Add Tags: Using tags allows you to add descriptive information to your setup. The descriptive information is useful when you plan to use more than one EFS setup with your network. Developers can also find the use of tags helpful when locating a particular setup to manipulate programmatically.

 

Choose Performance Mode: Changing the performance mode enables EFS to read files faster, but at the cost of higher latency (time to find the file) when using the Max I/O setting.

 

Amazon chooses the General Purpose setting by default, which means that transfer rates and file latency receive equal treatment when determining overall speed.

 

7. Perform any required optional setup and then click Next Step.

 

8. Verify the settings and then click Create File System.

The File Systems page appears with the new file system that you created. Even though the file system is accessible at this point, the drive may not be available immediately.

 

Creating additional file systems

AWS_file

 

 

Update the EC2 instance

Update EC2 instance

Before making any major change to your EC2 instance, it pays to perform an update to ensure that you have the latest versions of any required products.

 

Performing this step each time you perform a major task may seem like too much work, but doing so will save time, effort, and troubleshooting in the long run. The following steps describe how to update your EC2 instance:

 

1. Type sudo yum update in the terminal window and press Enter.

You see messages telling you that EC2 is installing any needed updates. If no updates exist, you see a No Packages Marked for Update message.

 

The sudo (superuser do) command makes you a superusersomeone who has rights to do anything on the computer. The yum (Yellowdog Updater Modified) command is the primary method for getting, installing, updating, deleting, and querying the operating system features. 

 

2. If EC2 performed any updates, type sudo reboot, and press Enter.

EC2 displays a message telling you that it has started the reboot, a process that stops and then restarts the system. You must close the terminal window.

 

3. Reconnect to the EC2 instance.

EC2 displays the normal terminal window. You generally don’t see any messages telling about the success of the update.

 

Installing the NFS client

NFS client

To work with EFS, you need the NFS client. The NFS client gives you access to specialized commands to perform tasks such as mounting a drive. You can optionally use the NFS client to interact with the EFS file system in other ways.

 

To install the NFS client, type sudo yum -y install nfs-utils and press Enter. (The yum –y command-line switch tells yum to assume that the answer to all yes/no questions is yes.)

 

If EC2 needs to install the NFS client, you see a series of installation messages. Otherwise, you see the following message in most cases:

 

[Note: You can free download the complete Office 365 and Office 2019 com setup Guide for here]

 

Performing the mounting process

mounting process

The EFS file system is ready for use and your EC2 instance has the correct software installed, but you still need to connect the two. It’s sort of like having a car and a tire to go with the car, but not having the tire on the car so that you can drive somewhere.

 

Mounting, the same process you’d use with that tire, is the step that comes next. 

 

1. Type sudo mkdir efs and press Enter.

This step creates a directory named efs using the mkdir (make directory) command for details on using the mkdir command). You can use other directory names, but using efs makes sense for this first foray into working with EFS. The next step involves a complicated-looking command.

 

In fact, unless you’re a Linux expert, you may have an incredibly hard time trying to decipher it. Fortunately, Amazon hides a truly useful bit of information from view, and the next step tells how to find it.

 

3. Highlight the command and paste it into your system’s clipboard.

 

4. Paste the command into MindTerm by clicking the middle mouse button (unless you have changed the default paste key to something else). 
sudo mount -t nfs4 -o nfsvers=4.1 $(curl -s http://169.254.169.254/latest/meta-data/placement/ availability-zone).fs-2b30c682.efs.us-west-2.amazonaws. com:/ efs

 

5. Press Enter.

EC2 mounts the EFS file system. Of course, you have no way of knowing that the EFS file system is actually mounted.

 

6. Type cat /proc/mounts and press Enter.

The entry tells you all about the EFS file system, such as the mounting point location at /home/ec2-user/efs.

 

Unmounting and removing the file system

directory

 

Working with the Elasticsearch Service

The data your organization creates manages, and monitors are the single most valuable resource that you own. Data defines your organization and helps you understand the direction your organization takes at various times during its evolution.

 

Consequently, you need to have applications, such as the Amazon Elastic-search Service, that can help you find what you need with as little effort as possible. The following sections help you discover more about the Amazon Elasticsearch Service so that you can use it to make working with your data substantially easier.

 

Understanding the Elasticsearch Service functionality

Elasticsearch Service

Organizations of all sizes need help in managing various kinds of data generated as part of application logging, application monitoring, the user clicks in web applications, and the results of user text searches (among a growing number of data sources). Elasticsearch (the application, not the service) is an open source product that helps you perform a number of tasks:

 

Searching: Locating specific pieces of information is hard, and the larger the drive, the harder the task becomes. Entire companies are devoted to the seemingly simple matter of helping you find a particular piece of information you need. You know that the information exists; you just can’t find it without help.

 

Analyzing: Data contains patterns, most of which aren’t obvious without help. Until you see the pattern, you don’t truly understand the data. Making sense of the data that appears on the hard drive is essential if you want to perform any useful tasks with it.

 

Filtering: Data storage objects today always have too much information. Anyone attempting to use the data gets buried beneath the mounds of useless data that doesn’t affect the current target. Sifting through the data manually is often impossible, so you need help filtering the data to obtain just the information you need.

 

Monitoring: Looking without actually seeing is a problem encountered in many areas, not just data storage. However, because data storage serves so many data sources, the problem can become especially critical.

 

Manually detecting particular data events, especially with a large setup, is impossible and there is no point in even attempting it. You need automation to make the task simpler.

 

Elasticsearch helps in these areas and many others. Here’s the idea: You begin with a large data source, such as the logs generated when users perform tasks using your application, and you need to make sense of all the data the data source contains.

 

When working with Elasticsearch, you mainly want to know about usage, which means performing analytics of various types.

 

To make these analytics happen, Elasticsearch creates indexes that help it locate data quickly so that it answers requests for information in near real time. You can use this information to perform tasks such as identifying outages and errors, even when users don’t directly report them 

 

Even though Elasticsearch is an open source product, it isn’t necessarily a complete product.

 

What Amazon is offering is a means to perform the required configuration details so that you can spend more time interacting with the data, rather than performing tasks such as creating data clusters and setting up redundancies to ensure that Elasticsearch works as intended. As with other services, you also gain access to the AWS security, scaling, and reliability features.

 

Many of these added features rely on Amazon CloudWatch to perform the required monitoring (the same as used by other services). Therefore, the Amazon Elasticsearch Service is a value-added proposition designed to make you more efficient. Fortunately, unlike your EFS setup, you won’t spend a lot of time using SSH to set up Elasticsearch, it comes with full console support.

 

As with many other Amazon services, Amazon doesn’t charge you for configuring the Amazon Elasticsearch Service. However, you do pay for the resources that Elastic-search needs to perform its work.

 

Therefore, you do pay for processing time and data storage access. You can find pricing information for this service at https:// aws.amazon.com/elasticsearch-service/pricing/.

 

Creating the Elasticsearch domain

Elastic_search

Before you can do anything with Elasticsearch, you must create a domain. The domain determines what Elasticsearch monitors. The following steps show you how to create a setup for the example EC2 system used in this blog:

1. Sign into AWS using your administrator account.

 

2.Navigate to the Elastic File System Management Console at https:// console.aws.amazon.com/efs.

You see a Welcome page that contains interesting information about the Amazon Elasticsearch Service and what it can do for you. However, you don’t see the actual console at this point

 

3. Click Get Started.

.You see the Create Elasticsearch Domain page. As shown in the figure, the first step is to give your domain a name.

 

4. Type my-domain and click Next.

Choose your domain name carefully. Amazon has some strict naming rules that it fully enforces. You see a Configure Cluster page, where you must define the characteristics of the cluster you want to use.

 

One of the most important settings for this example is the Instance Type. To keep costs as low as possible (conceivably free), you must select one of the free instance types.

 

5. Choose the t2.micro.elasticsearch option in the Instance Type field.

 

6. Scroll down to the Storage Configuration section of the page and choose EBS in the Storage Type field.

Amazon automatically configures the other required settings. You must use EBS storage when working with the free instance types.

 

7. Click Next.

You see the Set-Up Access Policy page, where you define who can access the Elasticsearch setup.

 

8. Choose the Allow Open Access to the Domain option.

Amazon automatically configures the access template for you. Normally Amazon doesn’t recommend using the open access option because it creates a potential security hole. However, in this case, the open setup makes experimenting with Elasticsearch easier.

 

9. Click Next.

You see a Review page, where you can review all the configuration settings you made. Each section provides an Edit button so that you can change specific settings without having to change them all.

 

10. Click Confirm and Create.

AWS creates the Elasticsearch domain for you. The Domain Status field tells you about the progress AWS makes in completing your setup. The Loading indicator shows that AWS is still working on the Elasticsearch domain. 

 

To complete this setup, you create a connection between the item you want to monitor using Elasticsearch and Amazon CloudTrail. For example, you can use Amazon CloudTrail to monitor log file output from your applications.

 

Understanding the AWS messaging services

AWS messaging services

Amazon SNS:

Amazon SNS, or Simple Notification Service, is asynchronous, managed service that provides the end user with the ability to deliver or send messages to one or more endpoints or clients. This works by using a Publisher–Subscriber-like model, as depicted in the following diagram:

 

One or more publishers or producers post a message to a corresponding SNS topic without knowing which subscribers or consumers will ultimately consume the message. The producer also doesn't wait for a response back from the consumers, thus making SNS a loosely-coupled service.

 

It is the consumer's task to subscribe to the topic and get notified of the incoming messages. SNS supports a variety of consumer implementation options, such as email, mobile push notifications or SMS, HTTP/HTTPS notifications, and even Lambda functions.

 

Amazon Kinesis:

Amazon Kinesis functions a lot like Amazon Simple Queue Service; however, it is fundamentally designed and optimized for high-throughput data writes and reads. Here, instead of a queue, you are provided with a stream that consumers can use to read from multiple times.

 

The stream is automatically trimmed after a span of 24 hours, so, unlike your consumers from the queue, here you are not required to delete the messages once they are processed:

 

Similar to Amazon Kinesis, AWS also provides a streaming functionality with DynamoDB as well, called DynamoDB streams. Using this feature, you can basically enable real-time changes to certain items within your tables in the form of a stream.

 

And, finally, you also get the standard request-reply model of messaging using a combination of Amazon API Gateway, ELBs, AWS Lambda, and other services.

 

This mode of communication is also synchronous in nature and can be used to fit a variety of use cases, as per your requirements. Keeping these basic differences in mind, let's now move forward and learn more about SNS.

 

AWS Lambda

Select the blueprint and fill out the necessary information for your function, such as its name, role name, and so on. Once done, from the SNS section, select the newly created SNS topic from the drop-down list.

 

Remember to select the Enable trigger checkbox before proceeding with the next steps.

 

Finally, in the Environment variables section, provide the appropriate values for the slack Channel and kmsEncryptedHookUrl parameters, as shown in the following screenshot.

Remember, the kmsEncryptedHookUrl is nothing but the Slack hook URL that we created a while back:

 

7. With the values filled in, simply select the Create function option and let the magic begin!

Based on the selected CloudWatch metric for your alarm, go ahead and create some synthetic load for your EC2 instance. Once the load crosses the set threshold in the alarm, it triggers a corresponding message to the SNS topic, which in turn triggers the Lambda function to post the alert over on the Slack channel.

 

In this way, you can also use the same SNS topic for subscribing to various other services, such as Amazon Simple Queue Service, for other processing requirements.

 

From the All metrics tab, filter and select the SNS option.

SNS option

Based on the requirements, you can now select between viewing the metrics based on the PhoneNumber, or Country, SMS type, and so on.

 

In this case, we have selected the PhoneNumber option to view the NumberOfNotificationsFailed and NumberOfNotificationsDelivered metrics.

 

Next, select the Graphed metrics tab to view the two metrics and their associated actions. Using the Actions column, select the Create alarm option for the metric that you wish to monitor.

 

Fill in the respective details and configure the alarm's threshold values based on your requirements. Once completed, click on Create Alarm to complete the process.

 

In this way, you can leverage Amazon CloudWatch to create and view logs and alerts generated by the SNS service. In the next section, we will be exploring and learning a bit about the second part of the AWS messaging services: Simple Queue Service.

 

Introducing Amazon Simple Queue Service

 

 

AWS regions

FIFO queues: When working with standard queues, there is a problem of maintaining the order of the messages and also ensuring that each message is processed only once.

 

To solve this issue, AWS introduced the FIFO queue that provides developers with a guaranteed order of delivery of messages, as well as the assurance that each message is delivered only once, where no duplicates or copies are ever sent out.

 

FIFO queues, on the other hand, do not offer an unlimited throughput capacity, unlike their predecessor. At the time of writing this blog, FIFO queues support up to 300 messages sent per second, with an additional 3,000 messages per second capacity if a batch of 10 messages per operation is performed.

 

Such queues are really useful when the order of the messages is of critical importance, for example, ensuring that a user follows the correct order of events while registering or purchasing of a product, and so on.

 

FIFO queues are currently only available in the US East (N. Virginia), US East (Ohio), US West (Oregon), and EU (Ireland) regions. With this basic understanding, let's look at some simple steps to get you started with your very own queue in a matter of minutes!

 

Creating your first queue

SQS queue

Getting started with your own Simple Queue Service queue is a fairly straightforward process. In this section, we will be looking at how you can create your very own standard queue using the AWS Management Console:

 

To begin with, log in to your AWS Management Console and filter out the Simple Queue Service service using the Filter option provided. Alternatively, you can also access the Simple Queue Service dashboard by selecting https://console.aws.amazon.com/Simple Queue Service/home.

 

Since this is our first time configuring the Simple Queue Service queue, select the Get started now option to continue.

 

Here, in the Create New Queue page, start off by providing a suitable name for your queue by filling in the Queue Name field. 

If you are building a FIFO queue, you will need to suffix .fifo after your queues name, for example, myQueue.fifo.

 

With the queue name filled out, the next step is to select the type of queue you wish to set up. In this case, let's first start off by selecting the Standard Queue option.

 

Next, select the Configure Queue option to go through some of the queue's configuration parameters. Alternatively, you can also select the Quick-Create Queue option to select all the default parameters for your queue.

 

In the Queue Attributes section, feel free to modify the following set of parameters for your queue, based on your requirements:

 

Default Visibility Timeout: Amazon Simple Queue Service does not automatically delete messages from the queue, even if they are processed by the consumers. Hence, it is the consumer's duty to delete the respective message from the queue after it has been received and processed.

 

However, due to the distributed nature of SQL, there is no guarantee that other consumers may not try to read from a copy of the same message.

 

To prevent such scenarios from occurring, Simple Queue Service sets a small Visibility Timeout period on a message once it is received by a consumer. This prevents other consumers from reading that message until the timeout expires.

 

By default, the Visibility Timeout can be set to a minimum of 30 seconds to a maximum of 12 hours. If by chance, the consumer is not able to process the message in the allocated timeout window, then the message will be delivered to another consumer and the process will continue until the message is not deleted from the queue by a consumer.

 

Message Retention Period: The amount of time Amazon Simple Queue Service retains a message in case it is not deleted. The accepted values here are a minimum of 1 minute and a maximum of 14 days.

 

Maximum Message Size: The maximum message size in bytes accepted by Amazon Simple Queue Service. The maximum limit is 256 KB.

 

Delivery Delay: Amazon Simple Queue Service allows you to temporarily delay the delivery of new messages in a queue for a specified amount of seconds. This is achieved by placing the new messages in a Delay queue which is completely managed by AWS itself.

 

Although it seems similar to the concept of Visibility Timeouts, a delay queue hides a message when it is first added to the queue, unlike the latter where the message is hidden when it is picked up by a consumer. The accepted values here are between 0 seconds and 15 minutes:

 

Receive Message Wait Time: Amazon Simple Queue Service periodically queries a small subset of the servers to determine if any new messages are available for consumption. This method is called short polling and is generally enabled by default when the Receive Message Wait Time is set to 0.

 

This method, however, results in a lot of empty responses as well, as sometimes messages just may not be present in the queue for consumption.

 

In that case, SQL also provides a concept of long polling, whereby Amazon Simple Queue Service waits until a message is available in the queue before sending a response.

 

This drastically reduces the number of empty responses and is helpful in reducing the overall running costs of your system. To enable long polling, simply change the value of Receive Message Wait Time to a value between 0 and 20 seconds.

 

With these basic settings configured, you can now go ahead and create your very own queue. Note, however, that there are a few additional settings that you can configure, such as a dead letter queue and server-side encryption. However, we will park these out for the time being. Select Create Queue once did.

 

With the new queue created, you can now start using it by simply copying the queue's URL (https://Simple Queue Service.us-east-1.amazonaws.com/<ACCOUNT_ID>/<QUEUE_NAME>) and providing the same to your applications or consumers to consume from:

 

You can also test the functionality of your queue by sending a text message to it using the Simple Queue Service dashboard itself. Select the newly-created queue from the Simple Queue Service dashboard, and from the Queue Actions, drop-down menu selects the Send a Message option.

 

This will bring up the Send a Message dialog box, as shown in the following screenshot. Next, type in a text message in the Message Body section and click on Send Message to complete the process.

 

You can optionally also change the delivery delay of this individual message by enabling the Delay delivery of this message by option and providing a value between 0 and 15 minutes.

 

With the message sent, you will be notified of the message's identifier along with an MD5 checksum of the body. Click on Close to close the Send a Message dialog box.

 

With this, the status of the Messages Available column should change to 1 as the new message is now waiting to be read or consumed. To read the message from the Simple Queue Service dashboard, once again select the Queue Actions drop-down menu and select the View/Delete Messages option.

 

This brings up the View/Delete Messages dialog box, as shown in the following screenshot. Here, the dialog will poll the queue once every 2 seconds until you have specified the polling to run using the Poll queue for the option.

 

You can also change the maximum number of messages viewed by modifying the View up to field. Once done, select the Start Polling for Messages option to get things underway:

 

With the polling started, you should see your text message in the display area, shortly. You can also verify the validity of the message by selecting the More Details option adjoining the message and verifying the MD5 checksum from the earlier recorded one.

 

Once completed, select the message and click on the Delete Messages option to remove the message from the queue. Remember, this is a permanent action and it cannot be undone. With the message deleted, your queue should once again show zero messages in flight or available.

 

Integrating Amazon SNS and Amazon Simple Queue Service

Amazon SNS SQS

One of the key features of Amazon Simple Queue Service is that it can easily be integrated with other AWS services, such as Amazon SNS. Why would I need something like that? To begin with, let us quickly recap the things we have learned so far about both SNS and Simple Queue Service:

 

Amazon SNS Amazon Simple Queue Service

  • Leverages the push
  • Leverages the polling mechanism

 

Amazon SNS messages can push messages to mobile devices or other subscribers directly Amazon Simple Queue Service needs a worker to poll the messages Persistence of messages is not supported Amazon Simple Queue Service supports message persistence which can come in really handy if you can't reach your consumers due to a network failure

 

From the table, it is easy to see that both the services offer their own pros and cons when it comes to working with them.

 

However, when we join the two services, you can actually leverage them to design and build massively scalable yet decoupled applications. One common architectural pattern that you can leverage by combining both SNS and Simple Queue Service is called the fan out the pattern.

 

In this pattern, a single message published to a particular SNS topic can be distributed to a number of Simple Queue Service queues in parallel. Thus, you can build highly-decoupled applications that take advantage of parallel and asynchronous processing. Consider a simple example to demonstrate this pattern.

 

A user uploads an image to his S3 bucket, which triggers an SNS notification to be sent to a particular SNS topic. This topic can be subscribed by a number of Simple Queue Service queues, each running a completely independent process from the other.

 

For example, one queue can be used to process the image's metadata while the other can be used to resize the image to a thumbnail, and so on. In this pattern, the queues can work independently of each other without even having to worry about whether or not the other completed its processing or not. Here is a representational figure of this pattern:

 

To integrate both the SNS and Simple Queue Service services, you will first be required to create a simple SNS topic of your own. Go ahead and create a new SNS topic using the AWS Management Console, as performed earlier in this blog.

 

Once the topic is ready, the next step involves the creation of an associated subscription. To do so, from the SNS dashboard, select the Subscriptions option from the navigation pane and click on Create subscription to get started.

In the Create subscription dialog box, copy and paste the newly created topic's ARN in the Topic ARN field.

 

Once the Topic ARN is pasted, select the Amazon Simple Queue Service option from the Protocol drop-down list, followed by pasting a queue's ARN in the Endpoint field. In this case, I'm using the standard queue's endpoint that we created a while back in this blog. With the required fields filled out, select Create a subscription to complete the process.

 

Next, from the Simple Queue Service dashboard, select the queue that you have identified for this integration and, from the Permissions tab, select Add a Permission to allow the SNS service to send messages to the queue. To do so, provide the following set of permissions:

1. Effect: Allow

Principal: Everybody

Actions: SendMessage

 

Once done, click on Add Permission to grant the SNS service the required set of permissions.

We are now ready to test the integration! To do so, simply fire a sample message using the Publish to Topic option from the SNS dashboard. Once the message is successfully sent, cross over to the Simple Queue Service dashboard and poll the queue using the View/Delete Messages option from under the Queue Actions drop-down list.

 

Here is a snippet of the Message Body obtained after long polling the queue: Similarly, you can use such a fan out pattern to design and build your very own highly scalable and decoupled cloud-ready applications.

 

Planning your next steps

Well, that was really quite a lot to learn and try out, but we are not done yet! There are still a few things that you ought to try on your own with SNS, as well as with SQL.

 

It's pretty easy and straightforward to get started with mobile push notifications. All you need is a set of credentials for connecting to one of the supported push notification services, a device token or registration ID for the mobile application and device itself, and an Amazon SNS configured to send push notification messages to the mobile endpoints.

 

You can read more about SNS mobile push notification services at https://docs.aws.amazon.com/sns/latest/dg/SNSMobilePush. html.

 

The other important feature worth trying out is the configuration of server-side encryption for your Amazon Simple Queue Service queue. You can leverage SSE to encrypt and protect data stored in your queue, however, this feature is only available in the US East (N. Virginia), US East (Ohio), and US West (Oregon) regions at present.

 

Encrypting the queue can be done at the time of the queue's creation, as well as after the queue has been created. Old messages present in the queue, however, are not encrypted if the SSE is switched on in an existing queue.

 

You can configure SSE for an existing queue simply by selecting it from the Simple Queue Service dashboard and selecting the Configure Queue option present in the Queue Actions drop-down menu. Here, check the Use SSE checkbox to enable the server-side encryption on your queue.

 

At this time, you will be prompted to select a customer master key (CMK) ID which you can leave to the default value if you do not have a CMK of your own.

 

Once done, set a duration for the Data key reuse period of between 1 minute and 24 hours. Click on Save changes to apply the recent modifications to the queue.

 

Last, but not the least, I also recommend that you try out the dead letter queue feature provided by Amazon Simple Queue Service. Dead letter queues are nothing more than queues that you create for storing messages that could not be processed by your application's main processing queue.

 

This comes in really handy when you need to debug issues in your application or the messaging system. However, it is very important to note that the dead letter queue of a standard queue is always a standard queue, and the same applies for a FIFO-based queue as well.

 

You can configure any queue within your account to be a dead letter queue for another queue by simply configuring the Redrive Policy of your application's main queue. 

Recommend