AWS Analytics Use Case - Data Pipeline and RedShift
By Peter Brammer
15th October 2019
Akkodis has been delivering data engineering and analytics services for several clients; In this post, we’ll speak about an industrial commercial organisation who needed assistance to understand the opportunities and implementation around their industrial data.
The Challenge
Our customer was challenged to be able to present a consolidated view of data from up to 3,000 industrial sensors that produce data every second from their manufacturing environment. The management of the organisation wished to view historical plant operations data in a precise and consistent manner.
Why Amazon Web Services
The company required a scalable cost-effective platform to securely store, process and visualise performance data from their manufacturing plant. The data needed to be encrypted at rest, during transmission, and eventually stored in a secure scalable data warehouse structure. Amazon S3 is a scalable, cost effective storage platform, proven for over 15 years to be durable, available, and horizontally scalable for high performance. More recently this is being used for massive data lakes – with data staged into suitable formats before being imported into data warehousing for querying.
Akkodis helped worked with our customer to architect, and deliver a managed pipeline for moving raw data with AWS Data Pipeline into S3, and then taking raw data, reformatting it using scale out processing on an Amazon Elastic Map Reduce, restaging this back to S3, and then importing it into Amazon RedShift to be ready to be efficiently queried. From here, visualisation dashboards permitted the client to interrogate their data.
Outcome and Results
By leveraging these managed components, Akkodis implemented a solution that kept direct costs and overheads as low as possible. Furthermore, the initial requirement for an hourly refresh was implemented with a 15 minute refresh cycle, giving the team higher confidence in the evidence of operation they were looking at.