IoT Big Data Value in Fleet Management

The Client

Our client is a technology company that develops telematics solutions for the transportation industry. Their products include sensors for trailers, chassis, and (shipping) containers that are integrated with the web application through IoT platform. The client provides real-time data, analytics, and insights to help fleet managers improve safety, reduce costs, and maximize asset utilization.

The client requested Akkodis assist in creating and operating an IoT service that would provide near real-time location intelligence of their fleet of vehicles and the items they are carrying.

The scope of considerations included:

  • Scalability — Scalability is a significant challenge for the Internet of Things (IoT) in the logistics industry. It requires robust connectivity infrastructure to handle the volume of data generated by connected devices, seamless integration of devices across locations, suppliers, and stakeholders, effective data management and processing, addressing security and privacy concerns, and efficient management and maintenance of physical infrastructure supporting IoT deployments. Implementing robust data management and analytics solutions, strong encryption, authentication mechanisms, and security protocols, and efficient deployment and maintenance processes are crucial for ensuring the scalability and reliability of IoT logistics systems.

  • Integration — The logistics industry faces challenges in integrating IoT due to the diverse systems and devices of various stakeholders. This requires standardization of data formats, protocols, and communication interfaces, as well as addressing compatibility issues between legacy systems and IoT technologies. Failure to integrate can lead to data silos, inefficiencies, and limited visibility.

  • Staff Skills — The logistics industry faces challenges in implementing IoT technologies due to the need for skilled staff. These staff must have deep understanding of network protocols, device connectivity, and data management. They must also possess cybersecurity knowledge to protect sensitive data. The evolution of IoT technologies demands staff to stay updated with emerging standards and protocols. Additionally, troubleshooting technical issues related to IoT devices requires strong technical skills, including identifying connectivity issues and resolving software glitches.

  • Data Quality — The logistics industry faces challenges in ensuring data quality in the context of the Internet of Things (IoT). The overwhelming volume of data generated by IoT devices, maintaining accuracy and reliability, and addressing data security and privacy concerns are crucial. Robust data validation and verification mechanisms are needed to address connectivity issues, power failures, and software glitches. Interoperability of IoT devices and systems is also a challenge, necessitating standardization and common protocols to maximize IoT benefits in logistics.

The data is delivered via Amazon Data Firehose (Kinesis) to various Amazon S3 buckets across multiple Amazon Web Services (AWS) accounts, organized by environment. Event notifications from these S3 buckets trigger Simple Notification Service (SNS) topics, which fan out events to Simple Queue Service (SQS) queues for further processing. An AWS Lambda function, triggered by SQS messages, processes, and transforms the JSON files, performs computations, updates Amazon DocumentDB, and publishes messages to an SNS topic for downstream consumers. The system also involves an Amazon OpenSearch domain for storing GPS locations, geofences, and event history, with plans to deprecate this functionality in favor of an API accessing data from an S3-based Datastore.

The back-end data processing aims to shift pagination, filtering, sorting, and searching logic from the front end to the back end, enhancing UI performance. This processing currently supports querying data from DocumentDB and OpenSearch. Each UI table fetches its data through a GraphQL query and a Lambda function that builds a DocumentDB aggregation pipeline. To maintain up-to-date table data, AWS CDC is used to capture and handle model changes, ensuring the table-data collection reflects current data from various DocumentDB collections. New tables require seeding existing data into the table-data collection using a dedicated Lambda handler and script.

  1. Data Management Across Accounts: Managing data flow from an external Kafka service to Kinesis Data Streams within AWS accounts, ensuring smooth and secure data transfers.
  2. Event Notification Handling: Efficiently managing S3 bucket notifications using SNS and SQS to handle various data processing requirements and prevent overloads.
  3. Real-time Processing: Ensuring near-real-time data processing and transformation, including error handling and retry mechanisms for Lambda functions.
  4. Data Consistency and Corrections: Maintaining data consistency by rectifying incorrect data formats and performing necessary computations before upserting into DocumentDB.
  5. System Scalability and Fault Tolerance: Implementing robust systems that can scale smoothly with increasing data volumes while providing fault tolerance and asynchronous processing capabilities.
  1. AWS Integration: Using Kinesis Data Firehose to seamlessly deliver data from Kinesis Data Streams to multiple S3 buckets across different environments.
  2. SNS and SQS for Notifications: Utilizing SNS for fan-out support and message filtering, and SQS to buffer events, decouple components, and manage retries and dead letter queues.
  3. Lambda Functions: Employing Lambda functions to process JSON files, perform transformations, corrections, and computations, and update DocumentDB with the latest sensor readings.
  4. DocumentDB and OpenSearch: Using DocumentDB for transactional data storage and the latest sensor readings, and OpenSearch for storing and querying GPS locations, geofences, and event history.
  5. Future-Proofing: Planning for deprecation of some functionalities and transitioning to API-based data fetching from S3-stored parquet files, ensuring the system remains scalable and adaptable.

The testing approach for verifying the end-to-end data flow from Kafka to Kinesis Data Stream, ensuring data is accurately delivered to S3 buckets via Kinesis Data Firehose. Tests to confirm that event notifications from S3 buckets trigger SNS topics and are correctly fanned out to SQS queues, enabling the Lambda function to process JSON files. The Lambda function’s ability to parse, correct, and transform data, perform runtime computations, and update DocumentDB. Additionally, tests ensured the publication of messages to an SNS topic and verify the functionality of DocumentDB and OpenSearch services, particularly for storing and retrieving GPS locations, geofences, and event histories.

The testing approach for back-end data processing includes verifying that all pagination, filtering, sorting, and searching logic is correctly handled on the back end. Tests ensured the Lambda functions build accurate DocumentDB aggregation pipelines and return the correct data to the front end. It’s crucial to confirm that data in the DocumentDB collections is up-to-date, handled by AWS CDC capturing model changes. For new tables, tests validated the seeding process, ensuring existing data is accurately inserted into the table-data collection without duplications. Additionally, verify that a single DocumentDB collection effectively aggregates data from different collections when applicable.

IoT-based telematics technology has revolutionized fleet management by providing real-time data and insights that optimize operations, enhance efficiency, and improve safety. Key benefits include fleet predictive maintenance, asset and cargo management, fuel management, route planning, driver safety, compliance with regulations, environmental impact reduction, customer service and communication, theft recovery and security, productivity improvement, and remote diagnostics.

IoT sensors can monitor vehicle health, asset and cargo location, fuel consumption, driver behavior, and idling, allowing for efficient scheduling and reducing downtime. Real-time tracking of vehicles, traffic conditions, and weather helps in route planning and optimization, while driver safety data can be used to implement training programs and reduce accident risks. Telematics also helps in environmental impact reduction, customer service, theft recovery, and productivity improvement.