At Re:Invent, Amazon Web Services offer new options for all phases of data in the cloud
Uploading data, ingesting data, getting insights from data — we typically associate all three capabilities with cloud workloads. Amazon’s Wednesday keynote announcements at Re:Invent unveiled new services for doing all of the above — with creative wrinkles all around.
Why sync data over the wire to Amazon, for instance, when you can instead mail it? And given how much data gets socked away in Amazon for analysis, how about a tool aimed at business folks, not IT personnel, for getting value from that data?
AWS Import/Export Snowball
“Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway,” computer scientist Andy Tannenbaum is reputed to have said. Amazon’s new appliance for migrating data to the cloud takes that notion to heart.
AWS Import/Export Snowball is a refined version of a service Amazon started offering back in 2009, where the user loaded data onto a device of their choosing and shipped it to Amazon with a manifest file. With Snowball, Amazon automates the process by providing the hardware to be loaded and streamlining the round-trip process.
The Snowball appliance is a ruggedized, tamper-proof, network-connected disk array, outfitted with a 10GB network port. The user fills it with up to 50TB of data at once, then ships it back to Amazon to have the data dumped into an S3 volume of their choosing. Each device costs $200 per job, with a daily penalty of $15 imposed for taking longer than the allotted 10 days to fill the device and ship it back.
Snowball’s appeal is meant to go beyond convenience, since many of its current and possible future features are aimed at assuring a customer its data won’t end up in the wild blue yonder. Not only is the data encrypted at rest on the device, Amazon can alert the customer whenever a Snowball job hits specific milestones: “in transit to customer,” “in transit to AWS,” “importing,” and so on. The e-ink status monitor on the front of the box even doubles as a shipping label, and Amazon mentioned the possibility of “other enhancements including continuous, GPS-powered chain-of-custody tracking.”
Amazon Database Migration Service
For those who are comfortable shuttling structured data over the wire incrementally into Amazon’s data centers, the company whipped the drapes off a similar item: Amazon Database Migration Service.
Users of Oracle, MySQL, or Microsoft SQL Server can replicate data from their data center to either the same database in AWS or convert it on the fly to a different one — for example, Oracle to MySQL. An included schema conversion tool ensures that the translated data won’t get mangled during the move, and Amazon claims it can suggest parallel ways to implement features that might not be available on the target platform.
Pricingis calculated by instance-hour for a virtual machine that runs the migration service (starting at 1.8 cents per hour), but data transfers to a database in the same availability zone as the Migration Service instance cost nothing.
Who’s the target audience? Most likely those looking to migrate to Amazon, but on their own terms and their own time. By setting up a path where data is passively replicated in the background, alongside existing business operations, they aren’t stuck in an all-at-once-or-nothing migration to AWS.
Amazon Kinesis Firehose
Amazon’s Kinesis was created to allow AWS customers to capture and work with live data, no matter the source. Its newest wrinkle, Amazon Kinesis Firehose, doesn’t expand on that idea. In fact, it cuts it down.
As the name implies, Firehose is little more than a connector that allows streaming data to be written into S3 or Redshift as it arrives. The only (optional) processing done on the stream in Firehose is compression or encryption, and the only options set by users are, for example, buffer size and the interval before data is delivered to its target bucket.
What’s interesting about Firehose is that it allows for data gathering and processing to be decoupled from each other. A user could, for instance, hitch an AWS Lambda job to trigger whenever Firehose data arrives in its target S3 bucket. That way, work can be done entirely on-demand as data arrives, using only as much code as is needed (per the Lambda processing model).
Data in the cloud isn’t much good on its own. Amazon has not traditionally lacked for options to collect and store data at scale, but now offers users a way to derive visualizations and insights from the bits they’ve amassed — through a service hosted right on Amazon.
Amazon’s new business intelligence product QuickSight connects to Amazon’s forest of existing database (RDS, DynamoDB, ElastiCache, Redshift) and analytics systems (EMR, Data Pipeline, Elasticsearch Service, Kinesis, Machine Learning). Once a given data source is connected, the user is presented with a UI akin to a simplified version of products like Tableau, with recommendations on what kinds of visualizations might be most appropriate for the selected data set.
Amazon’s cloud products are notoriously obtuse for users, but the interface for QuickSight seems straightforward and uncluttered — after all, it’s meant for the business side of an enterprise rather than IT. In another nod to business users is a promised feature where data harvested through QuickSight can be accessed via a SQL-like command language, so partner products (right now, Domo, Qlik, Tableau, and Tibco) can eventually make use of QuickSight’s in-memory processing. That said, there must be a better way to hitch Excel up to QuickSight, or Amazon will miss out on taking advantage of the single biggest self-service data tool in use in enterprises.
In another appeal to business users, QuickSight will cost $12 per user per month, or $9 per user for a year at a time. Up to 10GB of data taken into QuickSight from other systems can be stored for free. However, it’ll be a while before Amazon customers can judge if this is an improvement over legacy BI — QuickSight isn’t scheduled to launch officially until “early 2016.”
- Published in Cloud Storage
If Amazon Web Service is becoming a nearly ubiquitous technology, what does that mean for the future of data and how companies work with Amazon moving forward?
Amazon has been grabbing headlines this month, from its recent earnings report, which demonstrated a major revenue lift despite an increased discounting strategy, to an expose on the company’s work culture that caused ripples across the Internet and around the water cooler.
While many people may have bristled reading about working at a company that has been described as “bruising,” if the company’s recent revenue numbers mean anything, working with Amazon Web Services has become almost unavoidable.
According to Arik Hesseldahl at Re/Code, one of the ways that Amazon has been able to grow revenue while also significantly cutting costs on products and services is the rapid rate at which Amazon Web Services is growing. One Amazon executive estimates that “AWS is adding enough capacity to serve a mini-Amazon (when Amazon was the size it was 10 years ago) every single day.” Though Amazon has only released estimates on how many AWS customers it has, it’s safe to assume based on these figures that there are a lot of them.
So, if AWS is becoming a nearly ubiquitous technology, what does that mean for the future of data and how companies work with Amazon moving forward?
1. More companies will be dependent on AWS, both directly and indirectly. While this might sound scary, it doesn’t have to be. AWS gives companies access to some of the most exhaustive user data available, which is why so many organizations have become reliant on it. The key is to make sure that you aren’t too reliant on it, which may require building additional safeguards and back-ups into your own systems to ensure that you aren’t completely derailed by any AWS disruptions. Even if you don’t think that you’re using AWS, it’s possible that you are indirectly through another third-party, so make sure that the vendors that you’re working with are transparent about the resources that they’re using.
2. There will be a greater need to adapt existing technologies to work hand in hand with AWS. In order to maximize efficiency and reduce unnecessary redundancies, it may be necessary to adjust your current internal technologies to work more smoothly with AWS. If you’re making a significant investment in AWS, it doesn’t make sense to also be financing an internal tool that is doing the same thing. Your internal continuous improvements processes should spot these redundancies that can be corrected relatively easily and cost-effectively through application modernization.
In short, AWS is not going away anytime soon, but that isn’t necessarily a bad thing. The reason that AWS has grown to be the giant that it is today is that it provides tremendous value to organizations, and for many the benefits of working with them outweigh the potential risks of relying too heavily on the technology. The key is finding the right way to work together (including storing local backups) and ensure that you’re getting the most out of the services.
- Published in Cloud Storage
Sales are growing fastest for companies that ship generic storage hardware to big cloud providers, IDC says
The cloud is where the action is in enterprise storage.
Sales are way up for little-known manufacturers that sell directly to big cloud companies like Google and Facebook, while the market for traditional external storage systems is shrinking, according to research company IDC.
Internet giants and service providers typically don’t use specialized storage platforms in their sprawling data centers. Instead, they buy vast amounts of capacity in the form of generic hardware that’s controlled by software. As users flock to cloud-based services, that’s a growing business.
Revenue for original design manufacturers that sell directly to hyperscale data-center operators grew by 25.8 percent to more than $1 billion in the second quarter, according to the latest global IDC report on enterprise storage systems. Overall industry revenue rose just 2.1 percent from last year’s second quarter, reaching $8.8 billion.
These so-called ODMs are low-profile vendors, many of them based in Taiwan, that do a lot of their business manufacturing hardware that’s sold under better known brand names. Examples include Quanta Computer and Wistron.
General enterprises aren’t buying many systems from these vendors, but the trends at work in hyperscale deployments are growing across the industry. Increasingly, the platform of choice for storage is a standard x86 server dedicated to storing data, according to IDC analyst Eric Sheppard. Sales of server-based storage rose 10 percent in the quarter to reach $2.1 billion.
Traditional external systems like SANs (storage area networks) are still the biggest part of the enterprise storage business, logging $5.7 billion in revenue for the quarter. But sales in this segment were down 3.9 percent.
The smarts that used to be built into dedicated external storage systems are now moving into overarching virtualization systems that aren’t tied to hardware, Sheppard said. The software, not the hardware, defines the storage architecture. Like computing power, storage can now be managed per virtual machine instead of per unit of storage, which can simplify management and reduce enterprise operating costs over the long term.
All these changes are just beginning to play out and should keep accelerating for the next five years, Sheppard said. “It’s very early days.”
The cloud and virtualization trends didn’t reshuffle the main players in the second quarter but may have influenced some of their results. EMC remained the biggest vendor by revenue with just over 19 percent of the market, followed by Hewlett-Packard with just over 16 percent. EMC, which sells newer technologies like solid-state and software-defined storage but is also deeply invested in traditional platforms, dropped 4 percent in revenue, IDC said.
Other hot trends in storage systems include the growth of startups selling all-flash arrays and the increasing popularity in China of homegrown vendors like Huawei Technologies, Sheppard said.
Overall demand for storage capacity continued to grow strongly, with 37 percent more capacity shipped in the quarter compared with a year earlier.
(IDC is a sister company of IDG, which owns IDG News Service.)
- Published in Cloud Storage