The requirement for big data analytics has become commonplace in most companies today. Data has always been an essential part of any business, but the need to extract useful information from that data has been on a steady rise in just the past decade.
With the amount of data coming in from numerous sources growing exponentially every day, it’s easy to understand why big data analytics can be a challenge. It’s not only about getting the information you need, it’s also about doing it in a way that will not exceed your budget. And then there’s security to think about, too.
As with anything data-related, there are other, simpler issues at hand—such as storing all that data coming in. At least this part has undergone a tremendous change with the introduction of public clouds like AWS, which allows for almost unlimited storage capabilities. More importantly, AWS offers managed services that can actually help us analyze data.
So let’s take a look at two such AWS services, and see which one would be a better fit for your business needs.
AWS Services for Big Data Analytics
AWS Redshift is a cloud-based data warehouse and analytics service offered by Amazon’s public cloud and based on PostgreSQL. It’s a column-oriented database designed to work with SQL-based business intelligence tools, providing data to users in real time.
AWS Redshift is also a fully managed petabyte-scale solution that allows for various use cases. One of the most common of these is business intelligence, as Redshift provides high-performance queries on petabytes of semi-structured and structured data—and does so very cost-efficiently.
More interestingly, AWS Redshift allows for operational analytics, so you can, for example, pull massive amounts of data from S3 (e.g., logs) and get real-time operational insights. You can also use Redshift for predictive analytics, whether to run your own machine learning models or to integrate it with other AWS services like SageMaker.
Redshift can even combine multiple data sources (structured data, semi-structured data, and unstructured data).
Fully Managed, Fast, Familiar
And the fact that Redshift is a fully managed service means there is almost no overhead required for maintenance. You let AWS take care of the Redshift cluster, while you focus on the data. If you require more storage capacity, you can scale Redshift as much as desired—a flexibility that separates cloud computing from the old days of bare metal hardware.
Not only will you get extra resources based on your needs, but the cluster will also scale down when necessary to keep costs as low as possible.AWS Redshift is fast and offers encryption for your data, plus, since it’s based on PostgreSQL, it provides familiarity, as all SQL queries are supported.
With Redshift, you can choose between on-demand and reserved instance pricing. On-demand pricing lets you pay by the hour (based on the instance type of your requirements) and is very flexible, as there are no upfront commitments.
Reserved instance pricing, on the other hand, has you commit to a 1- or 3-year contract, with no upfront, partial upfront, or all upfront options and associated discounts. This can reduce your pricing by up to 75% and is an absolute must-have when you’ve already tested everything and know exactly what hardware is needed to handle your required tasks.
AWS Athena works differently compared to AWS Redshift, as it is not a data warehouse but an interactive query service. With it, you can easily analyze data that was previously stored in S3 buckets, plus it is a serverless product, meaning there is no infrastructure to manage or even think about. Athena is also very easy to use, as all you have to do is choose the data, define the schema to be used, and start with the queries using SQL.
Fast Analytics on Simple Queries
There are a few visible benefits of using Athena right from the start. For example, you don’t need to think about data and whether it is prepared or not—you simply start querying. Also, Athena is very fast, with results returned sometimes within seconds.
Athena is used for various kinds of data analytics—ad-hoc analytics, some type of streaming analytics, data lake analytics, etc. But where AWS Redshift is designed with complex queries coming from multiple sources in mind, Athena is used more for the simple queries run on a single data source.
As AWS Athena pulls data from S3 buckets, security is something to think about as well. Thankfully, Athena provides both encryption at rest (with server-side encryption, client-side encryption, and KMS encryption in place) and encryption in transit (TLS-level encryption is used for transit between S3 and Athena, and KMS is supported to encrypt various data sets from Athena query results).
When it comes to AWS Athena, there are some limitations that apply. For example, it can only support one query and have five concurrent queries running for each account. Additionally, Athena does not support queries across all available regions, so make sure you are aware of the location of your data before you start using it.
AWS Athena is priced only for the data scanned by the query you’re performing. This can be a significant advantage, as you don’t need to have a cost strategy in place. The cost of using Athena in the US-East-1 region (North Virginia) is $5.00 per TB of data scanned.
But do note that by compressing, partitioning, and converting your data into columnar formats, you can achieve not only significant savings on Athena queries, but also better performance.
AWS Redshift and AWS Athena Comparison Chart
|AWS Redshift||AWS Athena|
|Service type||Fully managed service, using nodes||Fully managed service, serverless model|
|Use cases||Business intelligence, data analytics; designed for complex queries for multiple data sources||Data analytics; designed for querying a single resource|
|Infrastructure||Runs on nodes within a cluster||Runs on serverless infrastructure, not visible or accessible to users|
|Pricing model||Cost per node used||Cost per amount of data scanned by the query|
|Data types used||Wide variety supported||Wide variety supported|
|Table performance and requirement||Uses PostgreSQL and is generally faster than AWS Athena||Uses Apache Hive Query Language (HQL)|
|Scaling capabilities||Resizing options available||Managed by AWS|
|Security||Data encryption supported||Data encryption supported|
|Performance||Depends on the node size and whether data is sorted||Depends on the queries you are performing|
|User-defined function (UDF) support||Supports UDFs with scalar and aggregate functions||Does not support UDFs|
N2WS Backup & Recovery
N2WS Backup & Recovery is a cloud-native product that can help you with your data backup needs. It has been around for a while now and is constantly evolving—with the 4.0 release even bringing support for Microsoft Azure Cloud.
Tracking Big Data Analytics in AWS
AWS Redshift and AWS Athena approach data analytics from different angles. Redshift is primarily a data warehouse service, which—depending on your business needs—can be a benefit or an unnecessary overhead.
- Athena is designed for data analytics and requires no data preparation nor carries additional costs for storage, as the data is already located in S3 buckets.
- AWS Redshift is overall faster when it comes to performance and offers more complex query capabilities, but that additional speed comes at the price of having various instances running (and being paid for).
All this means that there are multiple factors to consider when trying to find the best solution for you. Query capabilities should steer your decision, but you also need to consider price. More importantly, your choice should depend on whether you’re already using Redshift (or even have data warehouse needs) or not.
Or, do you want to spend additional time working with a full data warehouse at all, instead of issuing simple queries that can do the job for you.If you’re not sure which way to go, your best option is to test them both. Athena is priced per query, so you can easily try it out. On the other hand, AWS Redshift provides a free 2-month trial option (for those creating their first cluster).