Our data engineering team built a scalable data pipeline that performs batch and stream processing of autonomous vehicle (AV) data with high speed and accuracy. To enable seamless analytics, AV performance monitoring, and operational metrics processing, we integrated an analytics platform that allows:
We reduced the time between uploading data from an AV to the system at the end of a shift and ingesting it into the platform from 24 hrs to an average of 15 min.
We reached the acceptance criteria in stream data processing of 1 minute between a vehicle and an expert to expedite AV system performance optimization cycles.
United States
Autonomous vehicles
Data engineering
Service provided:
The client is a self-driving car system supplier company that builds autonomous vehicles for cities. Their service is organized into shifts on defined routes in several cities. After an AV finishes its shift, it returns to the garage and uploads logs from its gauges, devices, sensors, and hardware to the system. Subsequently, the information uploaded is used by data analysts, data scientists, ML engineers, autonomy engineers, robotics engineers, and others to improve their vehicles continuously.
Although the company had an in-house ETL to make the data available to its end users, it had significant flaws that forced it to invest in a new data pipeline. The main problem of the existing data infrastructure was its sequential operation, which handled one log at a time. It took at least one day to make the data available for further usage, i.e. ad-hoc analysis, reporting, ML, simulation, etc. The company’s use cases include real-time event detection, previously done manually.
Reaching ~100% vehicle autonomy
Ensuring prompt data ingestion
Cutting high infrastructure costs
The new data pipeline reduces the time needed to process and prepare AV data for further work. The data infrastructure has been modernized, and distributed data processing capabilities have been achieved. Scalability issues and delays in data processing have been cut down. Instead, the following capabilities were gained:
The company is interested in analyzing data coming from their AVs. This data provides valuable insights into their vehicles’ performance, safety, and efficiency, which is required for further work of autonomy engineers and robotics engineers. A non-exhaustive list of analyzed data sets is provided as follows:
The developed data pipeline has significantly sped up data ingestion and availability for end users, which enhanced the ability to do prompt analysis, reporting, and system optimizations.
The new architecture allows the system to easily accommodate increasing data volumes without compromising performance, growing infrastructure costs, or maintenance needs.
There is no longer a need for manually filled Excel spreadsheets while detecting real-time events since the pipeline automates calculations during streaming.
The availability of timely and accurate business-ready data and modern dashboarding capabilities enabled informed decision-making, strategic planning, and operational responsiveness.
The analytics platform provided has streamlined data operations, which leads to smoother and faster workflows, achieving an excellent level of cooperation, solid security, and convenience in data governance.
GreenM brings both deep expertise and a highly effective development team to every project they work on. In my time working with GreenM at NRCHealth, they not only delivered every project to spec and on time, but also elevated the level of our whole engineering department with their organizational and architectural best practices.
Great communication, fantastic partner, really smart about data and health data in particular. Senior Management are some of the best technical people I’ve ever worked with in more than 13 years. They consistently exceed expectations.
GreenM team has a lot of experience with AWS. They have deployed several solutions. Their knowledge is up to date and I’d highly recommend them to anyone who needs to build BI/analytics leveraging AWS.
We have worked with Alexey and the team at GreenM on many projects and have consistently been impressed with the quality of their work. They hire very highly skilled individuals and strive to understand not just our immediate needs but the underlying issues and how we can improve the process.
I’ve leveraged technical help from GreenM on numerous consulting projects from basic AWS setup and administration to implementing complex design using serverless managed AWS services for rapid development of scalable solutions to clients. GreenM has always delivered on-time and is a great partner to collaborate with.
GreenM is Starschema’s key partner from 2021. GreenM provided its services at a time when the market was looking for the most talented resources who are not only experienced but can also quickly manage the constantly changing technology world. GreenM quickly adapted to the Starschema working culture and high standards, and delivered technical professionals who could blend in easily. GreenM is a highly recommended partner for supporting the growth of any technical company with highly skilled and motivated professionals.
Copyright © 2024 GreenM, Inc. All rights reserved.
