Validate Health is a Chicago-based healthcare analytics firm. It helps providers and accountable care organizations (ACOs) manage financial risk and improve patient outcomes as they move to value-based care. In 2025, it was acquired by Oliver Wyman, part of Marsh McLennan, to join its actuarial consulting practice.

For a company like this, the models are the product. One of the most important is knowing how much each patient will cost. That's why we worked with their team across the full cycle of data science on healthcare data in Databricks: exploratory analysis, defining patient cost prediction models, building the architecture to evaluate and improve them, and deploying to production with MLOps. The result: we went from a baseline model with 50% reliability to a production model at 87.8%.

In healthcare analytics, a model isn't useful just because it "works in the notebook." It has to be accurate, explainable, and stable, because its outputs drive major financial and clinical decisions. The challenge had several layers:

• Complex, sensitive data: high-volume health data with many variables, missing values, and outliers, all requiring careful privacy handling.
• Choosing what to model: deciding which models and architectures make sense before investing in training them.
• Measuring what matters: knowing whether a new model version is actually better than the last one, not just different.
• Reaching production: moving from experimentation to deployed, versioned, and monitored models.

1. Exploratory data analysis

Before building any model, we got to know the data. Through exploratory data analysis (EDA) in Databricks, we reviewed distributions, quality, outliers, and relationships between variables. We identified the patterns and risks the model needed to account for. That foundation keeps you from building on wrong assumptions.

2. Model definition and deployment architecture

With those insights, we defined the patient cost prediction models and how they would be deployed: what problem each one solves, what data it trains on, how often it's updated, and how its results reach the people who use them.

3. Evaluation architecture: baseline model, evaluator, tests, and metrics

The core of the project was an architecture that lets you improve models based on evidence:

• Baseline model: a simple, well-understood reference model that sets the starting point. Ours started at 50% reliability, and every new version had to beat it to earn its place.
• Evaluator: an automated process that compares each candidate model against the baseline using the same rules and the same data.
• Tests: validation on data the model had never seen and across different historical periods, to confirm performance holds over time and across segments.
• Metrics: technical and business indicators that show, without ambiguity, whether a version is better.

4. Production deployment with MLOps

Models that pass the evaluator go to production with MLOps on Databricks:

• MLflow: every experiment is logged with its data, parameters, and metrics, so any result can be reproduced and compared.
• Unity Catalog: a central model registry with version control, traceability, and governed access permissions. This is key when working with health data.
• Model Serving: the production model is consumed via API, so Validate Health's applications and analysts get each patient's cost prediction when they need it.
• Monitoring: performance tracking to detect when the model starts to degrade and needs retraining.

A model learns by minimizing a loss function: the rule that tells it how bad each error is. Choosing it is a business decision, not just a technical one:

• Not all errors cost the same: when predicting patient cost, underestimating a complex patient can be far worse than overestimating. The loss function teaches the model which errors to avoid most.
• Outliers matter, but shouldn't dominate: a small number of patients can account for a large share of total cost. We chose loss functions that account for them without distorting predictions for everyone else.
• The model optimizes what you ask for: if the loss doesn't reflect business priorities, the model will be technically sound and commercially useless.

In short, the loss function translates "what hurts us most" into the model's language.

Good measurement is what turns an experiment into an asset. We evaluated each model on three levels:

1. Accuracy: how far predictions are from reality, on average and for the most expensive cases.
2. Stability: whether the model behaves the same across time periods and patient segments, or only works for some.
3. Business impact: whether technical gains translate into better risk and cost decisions.

That's how we went from 50% reliability in the baseline model to 87.8% in production, a 37.8-point improvement measured with the same rules at every step. With these metrics, every improvement cycle has a clear outcome: the new version either replaces the production model or gets discarded, and you always know why.

1. Sensitive health data: working with patient information under strict privacy and access controls.
2. Heterogeneous data with outliers: building useful features without losing the signal from the most relevant cases.
3. Fair comparisons: making sure every version was evaluated with the same rules as the baseline model.
4. From experiment to production: moving models into a versioned, monitored production environment.

• A deep understanding of the data through exploratory analysis, before investing in models.
• Patient cost prediction models with a defined deployment architecture.
• From 50% to 87.8% reliability: a 37.8-point improvement over the baseline model, measured with an evaluation architecture built on an evaluator, tests, and metrics.
• Production models with MLflow, Unity Catalog, and Model Serving, versioned, governed, and consumed via API.

Our work with Validate Health sums up our method:

1. Understand the data first: exploratory analysis defines what's possible and what isn't.
2. Set a baseline model: without a reference point, there's no way to know if you're improving.
3. Choose the loss function based on the business: the model should avoid the errors that cost the most.
4. Evaluate with technical and business metrics: an improvement only counts if it shows up in decisions.
5. Deploy with MLOps: log with MLflow, govern with Unity Catalog, serve via API, and monitor so the model stays reliable.

• MLOps / LLMOps: take models and agents to production and keep them reliable, measurable, and cost-efficient.
• AI Agent Audit and Evaluation: the same baseline, evaluator, and metrics discipline, applied to generative AI systems.
• Data for AI: data platforms on Databricks, Snowflake, or BigQuery, ready for analytics and machine learning.
• AI Consulting and ROI: decide what to model and how to measure its impact before you invest.

Do you have models that never leave the notebook, or that no one can say are getting better? Let's talk.