A practical guide to building enterprise-grade RL agents under strict privacy constraints

Image generated by Microsoft Copilot

This article highlights an approach we developed for improving an enterprise agent’s ability to assist users effectively. At Microsoft, we’ve built multiple copilot and agent products, each with their own strengths and capabilities. Different teams across our organization have experimented with various agent training strategies.

Related to developing and deploying these agents is the concept of reinforcement learning (RL), a method of Machine Learning that teaches machines to make decisions through interaction and feedback, similar to how young children learn. The content of this article focuses on lessons we’ve learned as we built RL pipelines for agents in an environment where we can’t use any user or organization content to train our models.

Our agents have been equipped with a suite of enterprise-grade skills including summarizing messages, understanding collaborative conversations, drafting follow-ups, and automating workflows. These capabilities are used by thousands of users daily. However, we identified a need to improve our agents’ ability to determine when to engage with users, trigger the appropriate skills, and continue with the correct workflow. This is because sometimes agents complete only the first step of a task before stopping, or do not trigger the right task (or if they do, they don’t follow through adequately), or they trigger too often.

If you’re facing similar challenges, especially in a data and privacy-constrained area, consider implementing these techniques that have proved valuable in our enterprise agent space:

• Implement offline reinforcement learning using zero real data (i.e., use synthetic and metadata only).
• Observe approximately 30 users to increase your sim-to-real correlation.
• Build a hierarchical policy architecture that enforces privacy at each level.
• Implement confidence-calibrated safety layers using Expected Calibration Error (ECE) thresholds.

We discuss these in detail below.

Explore simpler options before turning to RL

We don’t recommend immediately starting with RL. Instead, we recommend that you first do what we did and thoroughly explore simpler options. Ones that we explored, and our reasons for not continuing with them, include the following:

• Rule-based orchestration: This achieved modest task success improvements for us but proved fragile and impossible to scale across workflows.
• Supervised classifier (XGBoost): We found that this showed decent precision but poor generalization across organizations.
• Imitation learning (behavior cloning): We found this to be limited by a lack of real user traces and goal understanding.

Although they did not work for us, they might actually work great for your situation, and so they are worth exploring. RL is a lot of work to undertake and maintain and so these “low-hanging fruit” may be helpful and enough for your use case. As you start by considering these more traditional Machine Learning approaches, we recommend being aware of some of their additional limitations that we discovered along the way:

• Delayed reward signals: We found that success can only be measured after multiple steps or significant delay, making it difficult for simpler supervised approaches to assign proper credit.
• Sequential decision-making across sessions: An agent’s success depends on making multiple coordinated decisions over time, creating a complex action space where stateless classifiers making independent predictions inevitably fail.
• Zero training on customer data: We do not seek to train our models on customer data. This is not merely a policy matter, but an architectural one, as we work to design systems to make such training technically impossible.

Reframing your agent as an actual agent, not just a chatbot

We found it helpful to redefine our system as a sequential decision system with zero content visibility, with the following directives:

• State: Limit to non-content metadata only, including session timestamps, feature activation counts, anonymized workflow patterns, and organization-level configuration settings.
• Action: Design actions to trigger a specific skill, ask a clarifying question, or do nothing.
• Reward: Work with extremely sparse and delayed signals like session continuation, feature usage, and abstract completion patterns containing no user-identifiable information.

Building a zero-data access RL agent: Our recommendations

Once you’ve framed the problem as a decision problem under strict data isolation, you can then build an RL-based orchestration layer that can observe metadata patterns and then plan, act, and learn from outcomes without any access to customer content.

Offline RL with synthetic data and Conservative Q-Learning

Not using customer data means training only on synthetic data and anonymized metadata signals. We found that Conservative Q-Learning (CQL) provides robust policy learning without involving customer content, specifically penalizing uncertainty and unsafe generalization. With this in mind, our recommendation is to evaluate policies using fully synthetic data that mimics statistical patterns without containing or being derived from actual user content.

Hierarchical policy architecture with data isolation guarantees

It’s important to choose a hierarchical approach not for complexity, but because it allows rigorous data isolation enforcement at every policy level:

• Top-level policy: Use only basic metadata (feature activation counts, session duration).
• Mid-level policy: Apply domain-specific constraints using anonymized workflow patterns.
• Low-level adapter: Operate exclusively on session-local context with technical safeguards that prevent data persistence.

Privacy-first reward modeling

Additionally, we believe in designing a reward system with privacy as the absolute priority:

• Use doubly robust estimators over completely anonymous usage metadata.
• Apply conservative bounds to prevent potential user identification.
• Implement extensive simulation testing to validate an approach that never requires customer data.

Your initial metrics will likely show modest but consistent improvements, which we recommend validating through controlled A/B tests before scaling.

Safety layer with confidence threshold and zero data access

Furthermore, we advocate for wrapping every policy output in calibrated confidence bounds:

• High confidence → Take action
• Low confidence → Clarify or do nothing
• Ambiguous → Default to inaction

For calibration, we suggest using Expected Calibration Error (ECE) as the primary metric, measuring the difference between predicted confidence and empirical accuracy across bins. This allows you to tune the system differently for different organization types — larger enterprises should typically receive more conservative default settings prioritizing precision over recall.

Building intelligence through synthetic simulation

To build simulated data for your RL pipelines, there are certain best practices you can follow:

1. Synthetic generation: Create fully synthetic workflow patterns; for example, by asking LLMs.
2. Validation: Undertake statistical comparison with anonymized metadata signals (not content) with A/B tests to compare policy performance on a simulator versus a small real-world pilot.
3. Transfer measurement: Explicitly measure sim-to-real transfer by training policies in fully synthetic environments and evaluating on small, controlled real-world cohorts (i.e., at 1 percent of typical traffic).

But the best practice to develop simulated data is: Talk to users! Or even better, ask to observe them!

We acknowledge that this can be slightly awkward for everybody, and we think it’s why few people do it. (Let me just look over your shoulder, or more likely, share your screen while you work, it’s not weird at all, don’t worry about it…) But we have found it to be incredibly helpful.

Here’s what we recommend: Observe at least 30 users (we found 30 to be just enough) interacting with your system. And this should not simply be some product manager or user researcher undertaking this — it is YOU, the researcher, who should. Doing it yourself provides an astonishingly complete picture of how to structure your simulations. You’ll see precisely where users want something but get stuck, how they react, when they quit, or when they think “maybe if I prompted it differently?” among other learnings.

You might think, “We have a million users — how could watching just 30 possibly be sufficient?” The key insight is that you’re not trying to simulate each individual user. What these 30 observations provide is the shape of the conversation and the patterns to follow with simulated data.

Then go out and build your simulations. Now you at least have a taste or sense of what good conversations look like. A sim-to-real correlation coefficient of 0.8 is an extremely good bar so don’t worry about trying to optimize more than that. If you do get to that, let us know because we’d love to understand more!

Results, challenges, and lessons learned

Here we discuss expectations around measured impact and ongoing challenges.

Measured impact you can expect

A well-implemented RL-based system should show meaningful improvements across several key metrics:

• Significant increase in multi-step task completion
• Substantial improvement in skill trigger precision
• Better trigger recall across observed states
• Notable reduction in user-reported “unwanted suggestions”

Ongoing challenges you’ll face

Modeling behavior effectively without content access will remain one of your hardest challenges. While you can generalize at the organization level using configuration settings and anonymous metadata, true personalization without content remains an open problem.

You could explore techniques like organization-level pattern learning (fully anonymized) and session-local contextual awareness (with no data persistence).

Other challenges you’re likely to encounter include:

• Cold-start problem for new organizations (requiring weeks of adaptation).
• Explaining the system’s decision-making to customers and internal stakeholders.
• Debugging failures when they occur due to zero visibility into content.

Key insights for your implementation

Our experience developing and deploying this system revealed several insights that may be valuable for your implementation:

• RL can work in enterprise-grade environments when implemented with stringent reward shaping as well as safety and privacy constraints.
• Your enterprise-grade AI is likely to need a hybrid architecture: Consider combining LLM prompting, RL, and some custom/rule-based logic. No single approach suffices for complex enterprise scenarios. As for RL for long-horizon trade-offs: Maintain policy learning for complex temporal decision-making. Regarding LLM prompting for in-the-moment judgment: Leverage language models more effectively for nuanced, in-context decisions, among other approaches.
• The complexity of a hierarchical approach is justified by the real constraints of enterprise environments that demand high precision, serve organizations with vastly different needs, and where failures are extremely costly in terms of user trust.

We continue to explore how to personalize agent behavior without persistent identifiers or content history, a very interesting but hard frontier that goes beyond today’s privacy-first embeddings.

Topics for future articles

In upcoming articles, we’ll address several critical areas:

1. Better explainability tools that don’t reference customer content.
2. Failure detection without content access.
3. Privacy-preserving testing.
4. Model drift monitoring: Detecting when agent behavior diverges from expectations without content visibility.

My team would be happy to hear from you. Please leave comments in the Comments field below.

Aditya Challapally is on .