Enterprise Encryption Platform
Replacing a costly vendor solution with a custom, Spark-native encryption engine to save $3M annually and cut data latency by over 90%.
Annual Vendor Cost
Before
$3,000,000
After
$0
Savings improvement
Data Decryption Latency
Before
High (Per-Cell API Call)
After
Low (Spark Native)
>90% Reduction improvement
Project Overview
General Motors' data platform was locked into an expensive third-party encryption vendor, costing the company $3M annually. As we migrated to the cloud, this legacy system became a critical performance bottleneck. Its architecture required a separate API call to decrypt every single cell of data, causing unacceptable latency for enterprise-scale processing in Spark. My challenge was clear: design and build a new encryption engine from the ground up that was secure, performant, and could break the costly vendor dependency.
I led this initiative by replacing the proprietary system with a flexible, open-source foundation. To meet GM's stringent enterprise security standards, I architected a custom deterministic algorithm built on top of the new library. This enhancement ensured every value was properly salted and encrypted with a unique, deterministic IV string, providing a robust security posture that was fully vetted and approved by GM's internal security team.
The new engine was built to be Spark-native, allowing it to integrate seamlessly and performantly within our Databricks environment. This architectural shift completely eliminated the per-cell API bottleneck. The project was a massive success, saving the company $3M in annual contract costs while simultaneously reducing data decryption latency by over 90%, establishing a new, cost-effective security standard for our cloud platform.