Integrating sophisticated AI solutions into traditional business processes is no easy task. This is what our client, the DAT Group, faced while transforming the art of used car valuation into a science. In addition to combining deep-rooted domain expertise with the precision of machine learning (ML), they had to integrate Explainable AI (XAI) to bring clarity, trust, and efficiency to their valuation process.
This case study illustrates the profound impact of AI transparency on decision-making – beyond mere technology adoption. Initially very reluctant to use a new and unfamiliar solution, our client’s business teams were supported by our XAI experts in understanding and adopting a data-driven solution for estimating the cost of used cars.
Context & Challenges regarding AI Adoption
DAT (Deutsche Automobil Treuhand GmbH) Group is an established international company operating in the automotive industry. For over nine decades, they have been at the forefront of providing data products and services, particularly focusing on the digital lifecycle of vehicles. A key aspect of their offerings includes the provision of price estimates for used cars, a service utilized by a diverse clientele ranging from insurance companies to original equipment manufacturers. This service, pivotal to their operations, hinges on the accurate and efficient valuation of used vehicles.
However, our client faced a significant challenge in modernizing their valuation process. Traditionally, their approach combined deep domain expertise with market data analysis, a method that, while thorough, was predominantly manual. Relying on manual processes posed a substantial limitation: it restricted the ability to scale and accelerate the information retrieval and analysis process.
DAT Group embarked on an ambitious AI roadmap, collaborating with us to automate these manual data processes. The goal was to develop an ML solution that could offer data-driven estimations of used car prices, thereby enabling the team to make real-time, informed decisions.
However, the journey was not without its hurdles. As the project progressed, we encountered a critical challenge: stakeholder buy-in was remarkably low. Despite the advanced capabilities of the ML model, we observed a pronounced reluctance among team members to utilize the model’s predictions in their workflows. This resistance stemmed from a lack of trust and understanding of the model’s decision-making process. The situation was succinctly encapsulated by a team member’s statement: “As long as I do not understand the decision-making process, I do not trust the estimates.” This sentiment highlighted a crucial gap in the adoption of the new system: the need for transparency and understanding in the new AI-driven approach.
“I found instances where the model did not meet my expectations. As long as I do not understand the decision-making process, I do not trust the estimates.”, Team member at DAT Group
Our Approach: Ensuring AI Adoption
To tackle our client’s team’s challenges and concerns, we identified a critical need for transparency and comprehension in the AI solution.
Comprehensive Stakeholder Engagement
To address the acceptance issues surrounding the ML model, our first step was to engage deeply with the stakeholders at DAT Group. This involved conducting group interviews with a wide array of team members, including domain experts and those who would be working directly with the model. Our objective was to uncover the root causes of their reluctance.
We discovered that the primary concerns centered around a lack of understanding of the model’s predictions and the factors influencing these estimates. This feedback was crucial, as it highlighted the need for a more transparent and explanatory approach in our model deployment.
Development of an Intuitive Dashboard
Armed with these insights, we focused on demystifying the model’s outputs. A key element of our approach was the development of a user-friendly dashboard featuring self-explanatory visualizations.
We used the SHAP (SHapley Additive exPlanations) library, a powerful tool in XAI. SHAP’s ability to compute and visually present Shapley values allowed us to demystify the model’s predictions. This approach enabled stakeholders to see how various features, such as model, year, and mileage, influenced the estimated prices of used cars. We incorporated local explanations for each prediction (Local Explanation vs. Global Explanation concept), a feature that allowed users to delve into specific cases, examining how individual car attributes uniquely influenced their estimated values, thereby providing a granular view of the model’s decision-making process.
The dashboard’s interactive visualizations were designed not only to inform but also to engage users, allowing them to explore and understand the model’s rationale through real-time scenarios.
Enhancing Change Management Through Education
To further bridge the gap between the model’s capabilities and stakeholder understanding, we organized a series of question-and-answer sessions. These sessions were designed to clarify any remaining doubts and to provide a comprehensive explanation of the model’s workings.
We presented the library visualizations in detail, ensuring that each stakeholder could interpret the results and understand the underlying calculations. This educational approach crucially improved model transparency and built the team’s confidence in their use.
Continuous Model Improvement
An unexpected yet valuable outcome of our approach was the identification and correction of certain biases in the model. The interactive sessions and advanced visualizations facilitated stakeholder understanding and provided us with insights into the model’s performance.
This led to improvements in the underlying data and the integration of XAI as a fundamental part of our model development lifecycle. By doing so, we aimed to preempt biases and ensure the ongoing quality and relevance of the models deployed.
Through this multifaceted approach, we not only addressed the immediate challenge of stakeholder buy-in but also laid a foundation for continuous improvement and adaptation of the ML models. This ensured that the models remained effective, relevant, and trusted tools in the hands of DAT Group’s decision-makers.
Key Benefits
Our client derived key advantages from our collaboration, showcasing the profound impact of enhanced model transparency and stakeholder trust in AI-driven solutions.
Full Adoption and Enhanced Confidence
The most significant benefit realized from our approach was the complete adoption of the ML model by DAT Group’s business teams. Initially met with skepticism, the model gradually gained trust among the stakeholders, primarily due to the increased transparency and understanding facilitated by our approach. This shift from reluctance to reliance marked a pivotal change in how the team interacted with and valued their AI-driven tool, leading to a more data-driven approach in their decision-making processes.
Improved Accuracy in Estimates
A direct outcome of this increased confidence was the noticeable improvement in the accuracy of used car price estimates. The team transitioned from relying on intuition and traditional methods to trusting data-driven insights. This shift enhanced the precision of their valuations and streamlined their workflow, making the process more efficient and reliable. DAT Group is now able to make more informed decisions, leading to better outcomes in their core business operations.
Enhanced Model Quality and Relevance
The interactive and explanatory nature of our approach had an unexpected yet highly beneficial impact on the model itself. Through the feedback and insights gathered during the explanation sessions, we were able to identify and rectify biases in the model. This continuous refinement process not only improved the current model’s accuracy but also set a precedent for future model development, ensuring that subsequent models would be more robust, relevant, and aligned with real-world scenarios.
Streamlined ML Development Lifecycle
Finally, our collaboration led to an overall enhancement of their ML development lifecycle. By incorporating XAI principles and fostering a culture of continuous feedback and improvement, we were able to achieve shorter and more efficient development cycles. Thus, we reduced the time-to-market for new models and ensured that these models were more closely tailored to the specific needs and nuances of DAT Group’s operations.
Team Involved in this XAI Project
The successful implementation of this project was the result of a collaborative effort by a dedicated and specialized team. This team was carefully assembled to address the unique challenges and objectives of the project, ensuring a blend of expertise in both data science and XAI.
The team was composed of a Data Scientist and an XAI Engineer, who played a pivotal role in the project. They brought a deep understanding of ML and a specialized focus on XAI. Their responsibilities included developing the ML model, integrating the SHAP library for explainability, and creating the intuitive dashboard for the stakeholders. Their expertise was crucial in building the model and making it transparent and understandable to non-technical team members.
Technologies Used to Ensure AI Adoption
The project’s success was significantly bolstered by the use of advanced technologies:
- SHAP (SHapley Additive exPlanations): Utilized to create visualizations that explained the pricing model’s predictions. These visualizations helped demystify the model’s decision-making process, making it easier for the DAT Group’s stakeholders to understand and trust the model’s estimates.
- Databricks: Employed for its powerful data processing capabilities. It enabled the team to efficiently handle and analyze the large volumes of data involved in the used car pricing model. Its collaborative features also facilitated seamless teamwork, allowing data scientists and analysts to work together effectively.
- MLFlow: Played a crucial role in managing the machine learning lifecycle in this project. It was used for tracking the various experiments conducted during the model development phase, ensuring reproducibility and facilitating the comparison of different model versions. MLFlow also streamlined the deployment process, making it easier to integrate the model into DAT Group’s existing systems.