Empowering field engineers with SLM-based AI copilots in disconnected mode
Welcome to an era of Small Language Models
By and
In several industry scenarios, such as operating within a factory with limited connectivity, deploying cloud-based copilots is not feasible. However, Small Language Models (SLMs) offer a significant opportunity to empower industry field engineers. These technologies are designed to enable personalized, context-aware support, streamline workflows, and boost productivity across various industries. This article highlights key advancements, practical implementations, and future directions, emphasizing the significant impact of SLMs in revolutionizing human support systems. It includes the following sections:
- Industry scenario
- Innovative GenAI solution on Microsoft Edge
- Do we need SLM fine-tuning? How can it help?
- SLM output evaluation
- Benefits and challenges
- Conclusion
1. Industry scenario
In today’s industrial landscape, field engineers and service operators play a crucial role in sectors like manufacturing, healthcare, logistics, and customer service. These professionals are responsible for maintaining and troubleshooting complex machinery, ensuring operational efficiency, and minimizing downtime. However, they often face challenges such as inefficient manual processes, outdated documentation, and limited tool support, which can lead to inconsistent service quality and reduced customer satisfaction. Additionally, working in disconnected or edge environments and managing complex human workload scenarios further complicates their tasks. The goal, as illustrated in this article, is to empower field engineers with intuitive, advanced applications that optimize operations, ensure consistent outcomes, and elevate service excellence and customer satisfaction.
By offering data-driven insights and recommendations, an SLM-based copilot supports decision-making processes, making it a valuable tool across different sectors. Additionally, such a copilot’s ability to operate in disconnected mode ensures that engineers have access to critical information regardless of network connectivity, reducing downtime and improving productivity.
Industry use cases
An AI copilot operating in disconnected mode can be applied in various scenarios across different industries, including the following ones described below.
Manufacturing
Field engineers can leverage an AI-based application to enhance operational efficiency and product quality. However, an SLM-based copilot can complement this by providing on-ground assistance to resolve equipment failures. Together, the AI-based application and SLM-based copilot can help reduce downtime, prevent costly breakdowns, and resolve issues more quickly [A]. Additionally, this industry copilot can be integrated into quality control processes to automate the inspection of products for defects, ensuring consistency and reducing labor costs [B]. These use cases highlight the adaptability and significant influence of SLMs in the manufacturing sector.
Potential use cases include:
- On-ground assistance: An SLM-based copilot can quickly support user queries and address doubts by referencing the machine manual and previous maintenance history to resolve equipment failures and ensure timely maintenance. While SLMs like the base Phi-4 may not be highly effective in analyzing raw time series data, they can utilize function calling to trigger machine data analysis or employ other approaches. The Phi-4 SLM model is designed to be efficient and operate with limited computational resources, making it suitable for on-device applications and deployable as a serverless API in Azure.
- Quality control: SLMs are a suitable solution for automating product inspections for defects using image recognition and natural language processing, ensuring consistency and reducing labor costs. The Phi-4 SLM model’s stated efficiency with limited computational resources makes it attractive for on-device applications.
Healthcare
Medical professionals working in remote areas can rely on an SLM-based copilot to access critical patient information and medical guidelines. This helps them to provide accurate and timely care even in locations with limited connectivity.
Potential use cases include:
- Remote patient consultations: Medical professionals in remote areas can use an SLM-based copilot to access critical patient information, which helps them to have the necessary data to make informed decisions and provide accurate care, even when far from traditional medical facilities. Additionally, it is important to emphasize the security and patient regulatory governance of disconnected SLMs, ensuring that sensitive information is protected and compliance with healthcare regulations is maintained.
- Access to medical guidelines: In locations with limited connectivity, healthcare providers can rely on an SLM-based copilot for access to up-to-date medical guidelines and protocols, helping them stay informed about the latest best practices and treatment options, ensuring a high level of care for their patients.
Logistics
In the logistics industry, customer support and issue resolution are critical for maintaining high levels of customer satisfaction. An SLM-based copilot can be integrated into the customer support system to provide real-time assistance and improve the efficiency of handling customer queries and issues.
Potential use cases include:
- Automated query handling: An SLM-based copilot can analyze unstructured customer inquiries and provide instant, relevant responses by extracting key information from the text. This helps in reducing response times and improving customer satisfaction.
- Sentiment Analysis: An SLM-based copilot can perform sentiment analysis on customer inquiries to gauge the emotional tone of the messages. This allows support agents to tailor their responses and provide empathetic and effective support.
Customer service
Service engineers can use an SLM-based copilot to access user history and service records while on site. This helps them to provide personalized and efficient service, enhancing customer satisfaction.
Potential use cases include:
- On-site service history access: Service engineers can use an SLM-based copilot to quickly access customer history and service records while on site, helping them to provide personalized and efficient service.
- Enhanced customer satisfaction: By having immediate access to detailed service records, engineers can resolve issues more effectively, leading to higher customer satisfaction and loyalty.
2. Innovative GenAI solution on Microsoft Edge
To address these scenarios, an innovative copilot powered by SLMs is introduced, which functions effectively in both disconnected and connected environments. This technology aims to enhance the efficiency, decision-making capabilities, and safety of field engineers through a versatile UX framework that can be adapted for various industries such as manufacturing, healthcare, telecom, logistics, customer support, and more. Here is a diagram of the current unified operations design utilizing Azure technologies.
3. Do we need SLM fine-tuning? How can it help?
Fine-tuning is crucial for tailoring a pre-trained language model to excel in specific industry use cases. By further training the model on a smaller, task-specific dataset, it can be optimized to perform better on targeted tasks or domains, enhancing its effectiveness in real-world applications [C, D].
Azure Prompt Flow fine-tuning: Azure Prompt Flow can be used for fine-tuning models. It offers a comprehensive platform for distributed training, adaptable logging, and model quantization, making it helpful for experimenting with and deploying SLMs [E].
Custom fine-tuning: Custom fine-tuning involves adapting the model to specific customer needs. This can be done using various techniques such as full fine-tuning, partial fine-tuning (e.g. LoRA, QLoRA, DoRA) and adapter-based fine-tuning [F].
Steps for fine-tuning
- Data collection: Gather a task-specific dataset.
- Data preparation: Cleanse, tokenize, and normalize the data.
- Training: Use appropriate training mechanisms and hyperparameters.
- Evaluation: Assess the model’s performance using relevant metrics.
- Deployment: Export the model for deployment and set up the environment.
Example script: Here’s a basic example script for fine-tuning an SLM using Azure ML and torchtune, a PyTorch library for authoring, fine-tuning, and experimenting in this space:
import torchtune
from azureml.core import Workspace, Experiment
# Set up Azure ML workspace
ws = Workspace.from_config()
experiment = Experiment(ws, 'SLM_Finetuning')
# Define the YAML recipe for torchtune
yaml_recipe = """
model:
name: 'microsoft/Phi-3-vision-128k-instruct'
task: 'text-classification'
dataset: 'your-dataset'
epochs: 3
batch_size: 16
learning_rate: 2e-5
training:
distributed: true
logging: true
quantization:
apply: true
method: 'dynamic'
"""
# Run the finetuning job
torchtune.run(yaml_recipe, experiment)# This masked template was generated using GitHub Copilot from the source code.
This script sets up an Azure ML workspace, defines a YAML recipe for torchtune, and runs the finetuning job. You can customize the YAML recipe based on your specific requirements. [G]
4. SLM output evaluation
Evaluating SLMs involves using several techniques to help ensure they can meet the desired performance criteria. Here are some key evaluation techniques and steps:
Direct Preference Optimization (DPO): This technique fine-tunes language models based on user preferences using tools like DPOTrainer from HuggingFace TRL.
Reinforcement Learning from Human Feedback (RLHF): This method refines models through feedback-driven reinforcement learning. [G]
Instruction fine-tuning via prompt engineering: This involves fine-tuning SLMs to perform better on specific tasks or domains. [H]
Azure Prompt Flow: Azure Prompt Flow provides a platform for evaluating SLMs [H]. It allows for the creation of custom evaluation flows and metrics tailored to specific tasks and objectives. Here are the steps to set up an evaluation pipeline using Azure Prompt Flow:
- Create an evaluation dataset: Use a JSON, CSV, or TSV file containing questions and ground truth.
- Deploy the model: Deploy your language model to an Azure Managed Online Endpoint for scalable and secure inference. [E]
- Create custom evaluation flows: Use Azure Prompt Flow to create or customize evaluation flows and metrics.
Example script: Here’s a basic example script to set up an evaluation pipeline using Azure ML and Prompt Flow:
from azureml.core import Workspace, Experiment
from azureml.pipeline.core import Pipeline
from azureml.pipeline.steps import PythonScriptStep
# Set up Azure ML workspace
ws = Workspace.from_config()
experiment = Experiment(ws, "SLM_Evaluation")
# Define the evaluation script
evaluation_script = PythonScriptStep(
name="Evaluate SLM",
script_name="evaluate_slm.py",
compute_target="cpu-cluster",
source_directory="scripts",
arguments=["--input_data", "data/evaluation_dataset.csv"]
)
# Create and run the pipeline
pipeline = Pipeline(workspace=ws, steps=[evaluation_script])
pipeline_run = experiment.submit(pipeline)
pipeline_run.wait_for_completion()This script sets up an Azure ML workspace, defines an evaluation script, and runs the evaluation pipeline. You can customize the evaluation script based on your specific requirements.
Evaluating SLMs and LLMs on custom data using Azure Prompt Flow involves creating an evaluation pipeline that objectively assesses each model’s efficacy on custom data. This pipeline helps discern performance differentials between large and small language models and ensures the selected model meets predefined business and technical thresholds. [E, I]
5. Benefits and challenges
Implementing an AI copilot in disconnected mode offers several benefits and poses certain challenges:
Benefits
- Increased efficiency: An SLM-based copilot can provide real-time assistance, reducing the time required to complete tasks and helping to improve overall productivity.
- Enhanced decision-making: By offering data-driven insights and recommendations, an SLM-based copilot can support better decision-making processes.
- Reduced downtime: The ability to operate in disconnected mode helps engineers have access to critical information regardless of network connectivity, minimizing downtime.
- Emotional support: An SLM-based copilot offers timely assistance, aiming to reduce some of the stress associated with complex tasks and providing emotional support to engineers.
Challenges
- Integration with existing systems: Integrating an SLM-based copilot with existing systems and workflows can be complex and may require significant effort.
- Data privacy and security: Ensuring the privacy and security of sensitive information accessed by an SLM-based copilot is crucial.
- User training and adoption: Engineers may require training to effectively use an SLM-based copilot, and there may be resistance to adopting new technologies.
- Maintenance and updates: Regular maintenance and updates are necessary to ensure that an SLM-based copilot remains effective and up to date with the latest information.
6. Conclusion
We believe that this article illustrates a notable advancement in the field of AI-driven support tools for field engineers. By addressing the specific challenges faced by these professionals, an SLM-based copilot has the potential to not only enhance their efficiency but also contribute to a safer and more productive working environment. As industries continue to evolve, the use of SLM-based AI copilots stands as a testament to the transformative potential of AI in empowering professionals and driving operational excellence.
Commitment to responsible AI
At Microsoft, we are guided by our and along with decades of research on AI, grounding, and privacy-preserving Machine Learning. A multidisciplinary team of researchers, engineers and policy experts reviews our AI systems for potential harms and mitigations — refining training data, filtering to limit harmful content, query- and result-blocking sensitive topics, and applying Microsoft technologies like Azure AIContent Safety, InterpretML and Fairlearn. We make it clear how the system makes decisions by noting limitations, linking to sources, and prompting users to review, fact-check, and adjust content based on subject-matter expertise.
Disclaimer
Microsoft believes that when you create technologies that can change the world, you must also ensure that the technology is used responsibly. Our goal is to develop and deploy AI that will have a beneficial impact and earn trust from society.
References
[A] , Artificial Intelligence, December 26, 2024.
[B] , Markovate, November 13, 2024
[C] Span Fine-tuning for Pre-trained Language Models.
[D] Fine-Tuning Small Language Models: Practical Recommendations. Fine-Tuning Small Language Models: Practical Recommendations | by Liana Napalkova, PhD | Medium, Apr 30, 2024
[E] , Techcommunity Microsoft, May 23, 2024
[F] , August 28, 2024
[G] A Survey of Reinforcement Learning from Human Feedback. , Dec. 22, 2023
[H] , Nov 04, 2024
[I] , May 16, 2024
