How to Improve LLM Performance with Human and AI Feedback on Amazon SageMaker

Large Language Models (LLMs) have demonstrated remarkable capabilities across a myriad of natural language processing tasks, from content generation to complex reasoning. However, achieving peak performance and ensuring alignment with specific user needs and safety guidelines remains a significant challenge. Traditional fine-tuning approaches often fall short in capturing the nuanced preferences and implicit knowledge that are critical for real-world applications. This article explores how integrating both human and AI-generated feedback into the LLM fine-tuning pipeline on Amazon SageMaker can dramatically enhance model performance, robustness, and alignment.

Context and Motivation

While pre-trained LLMs offer a powerful foundation, their general nature often necessitates further adaptation to achieve optimal results for domain-specific or task-specific applications. Supervised fine-tuning (SFT) on curated datasets is a common initial step, but even high-quality datasets can struggle to cover the vast space of possible inputs and desired outputs. This is where feedback loops become indispensable.

The current state of the art in aligning LLMs often involves Reinforcement Learning from Human Feedback (RLHF). RLHF has revolutionized LLM training by enabling models to learn from human preferences, leading to more helpful, honest, and harmless outputs. However, RLHF itself presents several limitations:

• Data Scarcity and Cost: Collecting high-quality human feedback is labor-intensive, time-consuming, and expensive. Scaling human annotation to cover a broad range of scenarios can be prohibitive.
• Subjectivity and Consistency: Human preferences can be subjective and inconsistent, leading to noisy reward signals. Ensuring inter-annotator agreement across diverse tasks is challenging.
• Evaluation Bottleneck: Evaluating LLM performance, particularly for open-ended generation, is complex. Traditional metrics like BLEU or ROUGE may not fully capture quality, relevance, or factual accuracy.

These limitations highlight the need for more efficient and scalable feedback mechanisms. By strategically combining human insights with scalable, automated AI-generated feedback, we can overcome these hurdles, accelerate the feedback loop, and continuously refine LLMs for superior performance.

Overview of Solution: Leveraging SageMaker for Scalable Feedback Loops

Amazon SageMaker provides a comprehensive platform for building, training, and deploying machine learning models at scale, making it an ideal environment for implementing sophisticated LLM feedback loops. SageMaker’s integrated services allow for seamless data management, distributed training, and flexible deployment options, catering to the unique demands of LLM development.

Our proposed solution integrates both human feedback, primarily through Amazon SageMaker Ground Truth, and synthetic AI feedback, which can involve techniques like model self-evaluation, ensemble-based scoring, or automated reward modeling. This hybrid approach aims to:

• Scale Feedback Collection: Augment limited human feedback with vast amounts of AI-generated signals.
• Improve Feedback Quality: Use AI to identify patterns and anomalies that humans might miss, or to pre-filter/prioritize tasks for human review.
• Accelerate Iteration: Reduce the time required to collect and process feedback, enabling faster model improvements.
• Enhance Model Alignment: Guide LLMs towards more desirable behaviors and outputs, closely aligning them with real-world user expectations.

Architecture Diagram: End-to-End LLM Feedback Loop

The following diagram illustrates the end-to-end architecture for an LLM fine-tuning and feedback loop on Amazon SageMaker.

AWS Component Breakdown:

• Amazon S3: Serves as the central data lake for raw data, processed datasets, model artifacts, and feedback data.
• Amazon SageMaker Processing: Used for data preprocessing, feature engineering, feedback data aggregation, and model evaluation.
• Amazon SageMaker Training: Orchestrates the training jobs for initial LLM fine-tuning, reward model training, and RLHF/DPO. This leverages distributed training capabilities for large models.
• Amazon SageMaker Ground Truth: Facilitates scalable and efficient human annotation for collecting preference data, comparative judgments, or quality ratings on LLM outputs.
• AWS Lambda: Can be used for lightweight, event-driven tasks such as triggering feedback generation, data transformations, or real-time scoring.
• AWS Step Functions: (Optional) For orchestrating complex multi-step pipelines involving different AWS services, ensuring robust workflow management.
• Amazon Bedrock: (Optional) Can be used to leverage foundational models for generating synthetic feedback (e.g., using a more capable model to evaluate a less capable one) or for rapid prototyping of LLM applications.
• Amazon CloudWatch: Monitors training job metrics, endpoint performance, and provides logging for debugging.

Implementation Details

Setting up a robust feedback loop involves several key stages, each leveraging SageMaker’s capabilities.

1. Initial LLM Fine-tuning Pipeline on SageMaker

The first step is to fine-tune a base LLM on a domain-specific dataset. SageMaker’s Estimator API provides a flexible way to launch training jobs.

import sagemaker
from sagemaker.huggingface import HuggingFace

# SageMaker session and role
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

# S3 URI for your fine-tuning dataset
train_data_uri = 's3://your-s3-bucket/training_data/'

# Define hyperparameters for fine-tuning
hyperparameters = {
    'model_name_or_path': 'google/flan-t5-base',
    'output_dir': '/opt/ml/model',
    'num_train_epochs': 3,
    'per_device_train_batch_size': 4,
    'learning_rate': 2e-5,
    'do_train': True,
    'do_eval': False,
    'save_steps': 500,
    'logging_steps': 100,
    'fp16': True, # Enable mixed precision training
}

# Define your SageMaker HuggingFace Estimator
# Uses a pre-built SageMaker Deep Learning Container (DLC)
huggingface_estimator = HuggingFace(
    entry_point='train.py', # Your training script
    source_dir='./scripts', # Directory containing train.py
    instance_type='ml.g4dn.xlarge', # Or ml.g5.2xlarge for larger models
    instance_count=1,
    role=role,
    transformers_version='4.26', # Specify the transformers version
    pytorch_version='1.13',      # Specify the PyTorch version
    py_version='py39',
    hyperparameters=hyperparameters,
    distribution={'pytorch': {'data_parallel': {'enabled': True}}} # For distributed training
)

# Fit the estimator to start the training job
huggingface_estimator.fit({'train': train_data_uri})

print(f"Fine-tuned model artifact: {huggingface_estimator.model_data}")

train.py (simplified example):

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, TrainingArguments, Trainer
from datasets import load_from_disk
import argparse
import os

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_name_or_path", type=str)
    parser.add_argument("--output_dir", type=str)
    parser.add_argument("--num_train_epochs", type=int)
    parser.add_argument("--per_device_train_batch_size", type=int)
    parser.add_argument("--learning_rate", type=float)
    parser.add_argument("--do_train", type=bool)
    # ... add other hyperparameters as needed

    args = parser.parse_args()

    # Load model and tokenizer
    model = AutoModelForSeq2SeqLM.from_pretrained(args.model_name_or_path)
    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)

    # Load dataset from SageMaker's input channel
    train_dataset = load_from_disk(os.environ["SM_CHANNEL_TRAIN"])

    # Define training arguments
    training_args = TrainingArguments(
        output_dir=args.output_dir,
        num_train_epochs=args.num_train_epochs,
        per_device_train_batch_size=args.per_device_train_batch_size,
        learning_rate=args.learning_rate,
        do_train=args.do_train,
        # ... other training arguments
        bf16=True if args.fp16 else False, # Use bf16 for G4/G5 instances
        save_strategy="steps",
        save_steps=args.save_steps,
        logging_steps=args.logging_steps,
        evaluation_strategy="no", # Eval usually done in a separate job for RLHF
    )

    # Create Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        tokenizer=tokenizer,
    )

    # Train the model
    trainer.train()

    # Save the model
    trainer.save_model(args.output_dir)

if __name__ == "__main__":
    main()

2. Incorporating Human Feedback with Amazon SageMaker Ground Truth

After initial fine-tuning, the LLM can generate responses that need human evaluation. SageMaker Ground Truth allows you to create custom labeling workflows to collect human preferences, for instance, by asking annotators to compare two LLM outputs or rate a single output’s quality.

Steps:

1. Prepare Input Data: Your input data (e.g., source prompts and LLM-generated responses) should be in S3.
2. Create a Manifest File: A manifest file (.jsonlines) points to your input data and can include metadata.
3. Define a Custom Labeling Workflow: Use the Ground Truth console or API to define the instructions, task layout, and annotation tools. For LLMs, this often involves a “comparison” or “text classification” task.
4. Launch a Labeling Job: Start the job, specifying the input data, output location, and the workforce (private, vendor, or public).

Example Manifest File for Preference Labeling:

{"source": "Write a short story about a cat.", "output_a": "Whiskers, a fluffy tabby...", "output_b": "Milo, a sleek black cat..."}
{"source": "Explain quantum entanglement.", "output_a": "It's like two coins...", "output_b": "Quantum entanglement is a phenomenon..."}

SageMaker Ground Truth Labeling Workflow (conceptual):

The labeling interface would typically present the source prompt and then output_a and output_b, asking the human annotator to choose which output is better or more helpful.

3. Incorporating AI-Generated Feedback

AI-generated feedback offers a scalable alternative or complement to human feedback. This can be implemented in several ways:

a. Automated Reward Modeling

A common approach in RLHF is to train a separate reward model that predicts a “score” or “preference” for an LLM’s output given a prompt. This model is initially trained on a small dataset of human preferences. Once trained, it can generate synthetic reward signals for vast quantities of LLM-generated text.

Reward Model Training Pipeline:

Conceptual reward_model_train.py:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import load_dataset
import torch
import os
import argparse

class RewardDataset(torch.utils.data.Dataset):
    def __init__(self, data_path, tokenizer):
        self.tokenizer = tokenizer
        # Load and process your preference data (e.g., from a CSV or JSON file)
        # Each entry should be a tuple of (prompt, chosen_response, rejected_response)
        self.data = self._load_and_process_data(data_path)

    def _load_and_process_data(self, data_path):
        # Example: Load from a simple JSONLines file where each line is {prompt, chosen, rejected}
        dataset = load_dataset("json", data_files=data_path, split="train")
        processed_data = []
        for item in dataset:
            prompt = item["prompt"]
            chosen = item["chosen"]
            rejected = item["rejected"]
            tokenized_chosen = self.tokenizer(prompt + chosen, truncation=True, max_length=512)
            tokenized_rejected = self.tokenizer(prompt + rejected, truncation=True, max_length=512)
            processed_data.append({
                "chosen_input_ids": tokenized_chosen["input_ids"],
                "chosen_attention_mask": tokenized_chosen["attention_mask"],
                "rejected_input_ids": tokenized_rejected["input_ids"],
                "rejected_attention_mask": tokenized_rejected["attention_mask"],
            })
        return processed_data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]

def collate_fn(batch):
    # Custom collate function to handle variable lengths and padding
    chosen_input_ids = [item["chosen_input_ids"] for item in batch]
    rejected_input_ids = [item["rejected_input_ids"] for item in batch]
    chosen_attention_mask = [item["chosen_attention_mask"] for item in batch]
    rejected_attention_mask = [item["rejected_attention_mask"] for item in batch]

    # Pad sequences
    chosen_input_ids = torch.nn.utils.rnn.pad_sequence(chosen_input_ids, batch_first=True, padding_value=tokenizer.pad_token_id)
    rejected_input_ids = torch.nn.utils.rnn.pad_sequence(rejected_input_ids, batch_first=True, padding_value=tokenizer.pad_token_id)
    chosen_attention_mask = torch.nn.utils.rnn.pad_sequence(chosen_attention_mask, batch_first=True, padding_value=0)
    rejected_attention_mask = torch.nn.utils.rnn.pad_sequence(rejected_attention_mask, batch_first=True, padding_value=0)

    return {
        "chosen_input_ids": chosen_input_ids,
        "rejected_input_ids": rejected_input_ids,
        "chosen_attention_mask": chosen_attention_mask,
        "rejected_attention_mask": rejected_attention_mask,
    }

class RewardTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        chosen_rewards = model(
            input_ids=inputs["chosen_input_ids"],
            attention_mask=inputs["chosen_attention_mask"]
        ).logits
        rejected_rewards = model(
            input_ids=inputs["rejected_input_ids"],
            attention_mask=inputs["rejected_attention_mask"]
        ).logits

        # Bradley-Terry loss
        loss = -torch.nn.functional.logsigmoid(chosen_rewards - rejected_rewards).mean()
        return (loss, {"chosen_rewards": chosen_rewards, "rejected_rewards": rejected_rewards}) if return_outputs else loss

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_name_or_path", type=str, default="gpt2") # Base model for reward model
    parser.add_argument("--output_dir", type=str)
    parser.add_argument("--data_path", type=str)
    parser.add_argument("--num_train_epochs", type=int, default=1)
    parser.add_argument("--per_device_train_batch_size", type=int, default=1)
    parser.add_argument("--learning_rate", type=float, default=1e-5)
    args = parser.parse_args()

    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
    # Add a padding token if the tokenizer does not have one
    if tokenizer.pad_token is None:
        tokenizer.add_special_tokens({'pad_token': '[PAD]'})
    model = AutoModelForSequenceClassification.from_pretrained(args.model_name_or_path, num_labels=1) # 1 for reward score
    model.resize_token_embeddings(len(tokenizer)) # Resize if new tokens added

    train_dataset = RewardDataset(os.environ["SM_CHANNEL_TRAIN"], tokenizer)

    training_args = TrainingArguments(
        output_dir=args.output_dir,
        num_train_epochs=args.num_train_epochs,
        per_device_train_batch_size=args.per_device_train_batch_size,
        learning_rate=args.learning_rate,
        logging_dir=f"{args.output_dir}/logs",
        logging_steps=100,
        save_strategy="epoch",
        dataloader_num_workers=4,
    )

    trainer = RewardTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        data_collator=collate_fn,
        tokenizer=tokenizer,
    )

    trainer.train()
    trainer.save_model(args.output_dir)

if __name__ == "__main__":
    main()

b. Prompt Scoring / Ensemble-based Evaluation

Another form of AI feedback involves using another, potentially more capable, LLM (or a specialized model) to score or evaluate the outputs of the primary LLM. This can be done via prompt engineering or by fine-tuning a small classifier.

Example: Using Bedrock for prompt scoring (conceptual):

import boto3

bedrock_runtime = boto3.client('bedrock-runtime')

def score_llm_response(prompt, llm_response, evaluator_model_id="anthropic.claude-v2"):
    """
    Uses a more capable LLM to score the quality of another LLM's response.
    """
    evaluation_prompt = f"""Evaluate the following response based on the given prompt.
    Prompt: {prompt}
    Response: {llm_response}

    Provide a score from 1 to 5, where 1 is very poor and 5 is excellent.
    Also, provide a brief justification for your score.

    Score:
    Justification:
    """
    
    # Example for Claude-v2
    body = {
        "prompt": f"\n\nHuman: {evaluation_prompt}\n\nAssistant:",
        "max_tokens_to_sample": 200,
        "temperature": 0.1,
    }
    response = bedrock_runtime.invoke_model(
        modelId=evaluator_model_id,
        contentType="application/json",
        accept="application/json",
        body=json.dumps(body)
    )
    
    response_body = json.loads(response['body'].read())
    text_response = response_body['completion']
    
    # Parse score and justification from the text_response
    # (Requires robust parsing logic)
    # For demonstration, assume a simple regex or string split
    score_match = re.search(r"Score: (\d)", text_response)
    score = int(score_match.group(1)) if score_match else None
    
    return score, text_response

# Example usage
# score, justification = score_llm_response("Tell me a joke.", "Why did the scarecrow win an award? Because he was outstanding in his field!")

4. Reinforcement Learning with Feedback (RLHF) or Direct Preference Optimization (DPO)

With the feedback data (human and/or AI-generated) and a trained reward model, you can now perform RLHF or DPO. SageMaker provides environments for distributed training, which is crucial for these compute-intensive methods.

SageMaker RLHF Integration Pipeline:

For RLHF, frameworks like Hugging Face’s trl library (Transformer Reinforcement Learning) are excellent choices. SageMaker supports custom containers, allowing you to bring your own environment with trl installed.

Conceptual PPO/DPO training script (inside SageMaker Estimator):

import os
import argparse
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from datasets import load_dataset

def main():
parser = argparse.ArgumentParser()
parser.add_argument("--base_llm_path", type=str)
parser.add_argument("--reward_model_path", type=str)
parser.add_argument("--output_dir", type=str)
# Add PPO/DPO specific hyperparameters
args = parser.parse_args()
# Load base LLM
tokenizer = AutoTokenizer.from_pretrained(args.base_llm_path)
if tokenizer.pad_token is None:
    tokenizer.add_special_tokens({'pad_token': tokenizer.eos_token}) # Or a specific pad token

# For PPO, use AutoModelForCausalLMWithValueHead
# For DPO, use AutoModelForCausalLM
model = AutoModelForCausalLMWithValueHead.from_pretrained(args.base_llm_path)
ref_model = AutoModelForCausalLM.from_pretrained(args.base_llm_path) # For PPO reference model

# Load reward model
reward_tokenizer = AutoTokenizer.from_pretrained(args.reward_model_path)
reward_model = AutoModelForSequenceClassification.from_pretrained(args.reward_model_path)

# Load or generate prompts for RLHF
# For demonstration, assume a simple dataset of prompts
prompts_dataset = load_dataset("json", data_files=os.environ["SM_CHANNEL_TRAIN"], split="train")

# Define PPO config
ppo_config = PPOConfig(
    learning_rate=args.learning_rate,
    log_freq=10,
    ppo_epochs=4,
    # ... other PPO parameters
)

# Initialize PPOTrainer (for PPO)
ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=model,
    ref_model=ref_model,
    tokenizer=tokenizer,
    dataset=prompts_dataset,
)

# Main PPO loop
for epoch, batch in enumerate(ppo_trainer.dataloader):
    prompts = batch["prompt"]
    query_tensors = tokenizer(prompts, return_tensors="pt").input_ids.to(ppo_trainer.accelerator.device)

    # Generate responses
    generation_kwargs = {
        "min_length": -1,
        "top_k": 0.0,
        "top_p": 1.0,
        "do_sample": True,
        "pad_token_id": tokenizer.pad_token_id,
        "max_new_tokens": 64,
    }
    response_tensors = ppo_trainer.generate(query_tensors, **generation_kwargs)
    responses = [tokenizer.decode(r.squeeze()) for r in response_tensors]

    # Get rewards from the reward model
    # You'll need to prepare inputs for the reward model and get its logits
    rewards = []
    for i in range(len(prompts)):
        # Concatenate prompt and response for reward model input
        reward_input = reward_tokenizer.encode(prompts[i] + responses[i], return_tensors="pt")
        reward_input = reward_input.to(reward_model.device)
        reward_score = reward_model(reward_input).logits.squeeze().detach().cpu().item()
        rewards.append(torch.tensor(reward_score))
    
    # PPO step
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
    ppo_trainer.log_stats(stats, batch, rewards)

# Save the final model
ppo_trainer.save_pretrained(args.output_dir)
if name == "main": main()

Performance Evaluation

Evaluating LLM performance after incorporating feedback is crucial to understanding the impact of the feedback loop. Beyond traditional NLP metrics, human evaluation metrics are paramount.

Metrics:

• BLEU/ROUGE: For summarization or translation tasks, still useful for fluency/overlap, but often insufficient for open-ended generation.
• Perplexity: Measures how well the model predicts a sample of text. Lower is generally better.
• Win Rate (A/B testing): The percentage of times one model’s output is preferred over another in human evaluations. This is a direct measure of alignment with human preferences.
• Factuality/Hallucination Rate: Human or automated checks for factual accuracy.
• Safety/Bias Metrics: Evaluation of model outputs for harmful, biased, or unethical content.
• Task-Specific Metrics: Depending on the application (e.g., accuracy for question answering, coherence for creative writing).

Comparison: Human vs AI Feedback Models (Conceptual)

Feature	Human Feedback	AI Feedback (e.g., Reward Model)
Source	Human annotators (Ground Truth)	Trained AI model, LLM self-evaluation
Scalability	Limited by human workforce, costly	Highly scalable, cost-effective for large datasets
Cost	High (labor, platform fees)	Lower (compute for training/inference)
Nuance	Captures subtle preferences, subjective judgment	Can be trained to mimic human nuance, but limited
Consistency	Can be inconsistent due to subjectivity	Generally consistent once trained
Speed	Slower (human processing time)	Faster (automated processing)
Bias	Inherits human biases	Inherits biases from training data and human feedback
Use Case	Gold standard for preference learning, complex tasks	Augment human data, accelerate feedback, early filtering

Sample Before/After Performance Table (Hypothetical):

Metric	Initial LLM (SFT)	LLM with Human Feedback (RLHF)	LLM with Human + AI Feedback (RLHF)
Perplexity (lower is better)	25.4	18.1	16.5
Win Rate (vs Baseline)	30%	65%	78%
Hallucination Rate	15%	8%	5%
Helpfulness Score (1-5)	3.2	4.1	4.5
Consistency Score (1-5)	3.0	3.8	4.2

This table demonstrates the incremental improvement achieved by introducing human feedback, and further gains when AI feedback is strategically combined.

Best Practices & Pitfalls

Best Practices:

• Iterative Approach: Start with a small human feedback loop to bootstrap the reward model, then scale with AI-generated feedback. Continuously iterate and refine both.
• Diverse Human Feedback: Ensure your human annotators represent your target user base to capture diverse preferences and minimize bias.
• Robust Reward Model Training: Train your reward model on a sufficiently large and diverse set of human preferences. Regularly re-evaluate and retrain it.
• Prompt Engineering for AI Feedback: When using LLMs for synthetic feedback, carefully craft prompts to elicit the desired evaluation criteria and minimize “model bias” from the evaluator LLM.
• Version Control: Meticulously track dataset versions, model artifacts, and training configurations using SageMaker’s experiment tracking or external tools.
• Automate Data Pipelines: Use AWS Step Functions and Lambda to automate the flow from data generation, feedback collection, training, and deployment.
• Monitor Model Drift: Continuously monitor the deployed LLM’s performance and output characteristics. If performance degrades, it may indicate concept drift or a need for re-training with fresh feedback.
• Data Validation: Implement strong data validation steps for both human and AI-generated feedback to catch errors or inconsistencies early.

Pitfalls:

• Reward Hacking: The LLM might learn to exploit flaws in the reward model, generating outputs that score highly but are not truly aligned with desired behavior. Regular human review helps mitigate this.
• Bias Amplification: If the initial human feedback contains biases, the reward model and subsequent LLM will amplify these biases.
• Overfitting to Feedback: The LLM might overfit to the feedback data, leading to a loss of generality or performance on unseen examples. Regular evaluation on held-out test sets is crucial.
• Misaligned AI Feedback: Poorly designed AI feedback mechanisms can lead to the model learning undesirable behaviors. Validate AI feedback against human judgments.
• Cost Overruns: RLHF can be computationally expensive. Use SageMaker’s managed Spot Training and efficient instance types (e.g., G5 instances) to optimize costs.
• Label Quality Issues: Inconsistent or low-quality human labels can poison the feedback loop. Invest in clear instructions, thorough training for annotators, and quality control mechanisms within Ground Truth.

Conclusion

The integration of human and AI feedback loops represents a paradigm shift in how we build and refine Large Language Models. By leveraging the comprehensive capabilities of Amazon SageMaker, ML engineers and data scientists can establish scalable, efficient, and robust pipelines for continuous LLM improvement. Human feedback provides the essential ground truth and nuanced preferences, while AI feedback offers the scalability and automation needed to accelerate the iterative development cycle. This hybrid approach enables LLMs to achieve higher levels of performance, better alignment with user intent, and greater robustness in real-world applications.

As LLMs become increasingly central to various products and services, the ability to rapidly and effectively incorporate feedback will be a critical differentiator. The architecture and best practices outlined in this article provide a strong foundation for Amazon engineers and external advanced ML users to harness the power of human and AI feedback on SageMaker, driving the next generation of intelligent applications.