人工智能
Indegene如何实现生命科学中企业数据质量的现代化和升级
此帖与来自Indegene Limited的Shravan K S共同撰写。
企业AI正在为生命科学组织创造新的机会，以改善商业、医疗、市场准入和其他商业职能的决策。 但每个AI模型的质量，仪表板，和商业见解，都取决于一件事：可信数据。 随着企业数据跨越多个平台、管道和业务领域的增长，维持一致的数据质量变得越来越困难。 人工数据质量规则的建立，零散的治理，以及不断演变的业务定义，使得整个企业的数据质量规模化具有挑战性。
本文探讨Indegene如何在其NEXT统一数据平台(NEXT UDP)内在AWS上构建生产级大型语言模型(LLM)为数据质量Agent提供动力。 agent生成上下文认知数据质量规则，验证，并在集中式，人行化治理下生成即时运行的SQL和Python代码。 这些能力共同帮助各组织提高企业规模的数据质量，同时减少建立和维持数据质量方案所需的努力。
关于 Indegene
Indegene Limited(BSE:544172,NSE:INDGN)是生命科学产业的战略运营伙伴，负责整个产品生命周期的商业，医疗，研发和安全运营。 Indegene受到世界前20家生物制药公司的信任，帮助在全球范围运行复杂、高度规范的工作流程。 Indegene将深层生命科学专业知识与AI-嵌入式平台结合，并规范操作模式，以提高任务关键业务的进度，一致性和合规性。 通过将技术纳入工作执行和规模，Indegene帮助生命科学组织在规范环境中取得可衡量的成果。
为什么常规数据质量程序会崩溃
大多数组织已建立了数据质量程序，但随着企业数据生态系统的发展，越来越难以有效保持这些程序。
最大的挑战之一是技术和商业规则所有权之间的鸿沟。 数据工程师通常定义技术规则，确认数据在结构上是否正确。 例如，验证HCP(健康护理专业ID)是否无效，是否遵循要求的格式，是否独特，业务团队制定规则，确定数据在业务上是否合理。 例如，确认产品市场在某一领土内所占的份额为100%，或确认一个方案主任的处方活动符合其专长和资格标准。 由于技术规则往往在代码中保留，而商业规则则包含词汇、标准业务程序以及主题事项专门知识，因此很难在数据集和平台之间保持统一和一致的管理。
传统的验证技术也有固有的局限性。 检查无效值、数据类型、独特性或偏好完整性仍然至关重要，验证产品等级、客户分割、地区调整或商业规则
建立和维持数据质量规则同样要求很高。 每个新的数据集都需要数据剖析，规则定义，SQL开发，验证，以及持续的维护。 这些规则往往分布在SQL脚本，管道代码，电子表格，数据目录，以及工具专用配置中。 随着计划、业务定义或验证阈值的改变，更新和管理这些规则所需的努力大大增加。
此外，业务影响很大。 由于许多活动都是手工重复的，所以登入新的数据集变得较慢。 维持数百个数据集的质量规则需要宝贵的工程努力。 业务用户在验证规则难以理解、追踪或与业务环境相关时失去信心。
因此，数据质量往往成为一种定期的治理活动，而不是一种持续、可扩展的能力。
用基因AI重新思考数据质量
传统数据质量工具往往作为独立能力运作，与更广泛的企业数据生态系统脱节。 随着组织规模化AI和分析，数据质量需要嵌入到数据平台本身，其中治理，元数据，线性，分析等部门合作。
Indegene通过NEXT统一数据平台(NEXT UDP)来解决这个问题，这是一个专门为生命科学而建立的现代，AI驱动的数据平台。 NEXT UDP 带来将分散的商业，医疗，临床和企业数据合并成一个单一的，有规范的环境，这样各组织就可以管理从摄入到分析的完整的数据生命周期。 NEXT UDP提供多个独立的AI agents，如数据摄入agent，数据质量agent, Orchestration agent，数据目录agent，以及SQL瓶式发电机agent. 您可以在您的数据管道中单独使用它们，或者进一步定制它们用于agent到agent的通信。
通过将数据摄入、转换、数据质量、治理、元数据管理和分析纳入一个统一的平台，NEXT UDP帮助各组织：
- 将分散的企业数据整合成一个可靠的真相来源。
- 实现各业务领域数据质量和治理标准化。
- 提高元数据的能见度、分布和可追踪性。
- 减少跨数据工程工作流程的人工工作。
- 为企业分析和决策提供可信赖的、已准备好的AI数据。
Indegene的生命科学客户需要agent，可以可靠地运行在生产上，并满足严格的治理要求，同时让团队灵活地构建。 在NEXT UDP内部，Generative AI赋予智能数据质量Agent以增强现有数据质量过程的权力。 agent通过加快规则创建，简化验证，帮助保持企业规模的一致治理，提升数据工程师和数据管理者生产率。
Indegene在一套AWS服务上建立了数据质量Agent，其核心是Amazon Bedrock,Strands Agents和Amazon Bedrock Agent Core. Amazon Bedrock，提供了基础模型(FM)的选择，建立在安全，隐私，以及负责的AI控制的基础上。 对于生命科学组织来说，这一点很重要，因为它们的数据往往包括敏感的保健专业和病人信息。 Strands Agents精简了agent的建设和更新，其评价工具帮助Indegene测试规则精度，这在规范的生命科学环境中至关重要。 agent运行于Amazon Bedrock Agent Core上，这是一个在规模上构建，连接和优化agents的平台，拥有任何框架或模型。 该解决方案还继承了现有的AWS安全和审计控制，因此Indegene可以专注于数据质量逻辑而不是agent基础设施。
是什么使得数据质量Agent独一无二？两个知识库，即能了解环境的数据质量
任何数据质量工具都可以检查无效值，重复或无效的数据类型。 AWS Glue Data Quality等服务，是一种AWS Glue的能力，已经从数据统计中推荐了这样的规则，它们仍然是坚实的基础。 更困难的问题是确定数据是否符合其业务目的。 为了解决这个问题，数据质量Agent使用两个互补的知识基础，将所编译的域知识与从规范使用中获得的经验结合起来。
第一个是内部企业知识库：它结合了内部业务词汇，数据词典，SOP，监管指导，批准的数据质量规则，审定的脚本，剖析结果，历史异常，以及主题专家(SME)的反馈。 内容通过检索增强生成(RAG)进行吸收、矢量化和检索，以生成基于企业标准和已证实结果的数据质量规则。
第二个是外部参考知识库：它使用核准的标准化数据质量框架、网络搜索和可控网络搜索来建立和定期更新。 当选定数据集时，agent分析表格和列元数据，并检索相关数据质量标准，参考值，以及来自可信来源的验证最佳做法。 例如，对于NPI(国家提供方标识符)一栏，它可以确定适用的质量标准，推荐格式(要求的10位数字格式)，并进行参考验证检查。
如何运作
数据质量Agent将用户从数据剖面到规则创建和治理在一个单一的工作流程中：
- 选择源：连接到Amazon Redshift或Snowflake等支持源。
- 选择数据库和计划：浏览可用的表格，列出每个表格的规则数和有效规则状态。
- 概况数据：该解决方案为表格计算质量度量和列模式，供用户审查。
- 检测异常：解决方案标出与历史基线的异常，用户审查总数、关键和警告问题以及受影响的记录。
- 管理规则：审查表格的现有规则或开始创建新的规则。
- 选择规则模式：选择列级、表级或自定义查询。
- 生成规则建议：agent使用元数据，剖析结果，异常，现有规则和知识库背景来推荐技术和业务规则，每条规则都具有严重性。
- 审查、完善和适用：用户编辑或核准每项建议。
- 生成和验证代码：agent为核准的规则生产SQL和Python，并有一个验证步骤检查它们。
- 存放在中央规则存放处：核准规则可供执行、监测和审计。
AWS上的基因AI动力数据质量Agent的解决方案架构
以下的图显示，即NEXT UDP平台上的数据质量Agent的端对端架构，从数据管理员请求到规则执行。
数据管理员和数据质量分析师从Indegene应用程序中调用解决方案。 下表提供了溶液管道的走行通道，并列出了每层使用的AWS服务。 它也捕捉了解决方案所支持的其他选项，这样你就可以根据自己的数据卷，现有工具，以及操作偏好来调整它。
数据质量 Agent 如何构建上下文
在建议任何验证规则之前，数据质量Agent从六个不同的来源集合上下文，每个来源都回答关于数据的不同问题：
这些来源一起让agent像经验丰富的数据管理员一样理性。 将每一项建议都建立在数据是什么、标准要求是什么、企业如何界定质量以及已经执行的内容上。
解决方案的作用
数据质量Agent生成数类检查，并让数据管理员控制结果。
异常检查生成
这个解决方案会产生异常检查，包括负切换百分数、市场份额不等于市场100%、药品排名不一致(例如，某药品排名第一，其份额低于药品排名第二)、设定比例表以外的评级值以及过于小或与答卷人数目不符的面板尺寸。
带有 RAG 的背景意识规则生成
数据质量Agent以三种方式生成规则建议：
创建新质量规则接口提供专用的列级，表级，以及自定义查询模式。
该平台支持技术验证和具体业务质量检查。
关于技术规则，agent
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
How Indegene is modernizing and scaling enterprise data quality in life sciences
This post is co-written with Shravan K S from Indegene Limited.
Enterprise AI is creating new opportunities for life sciences organizations to improve decision-making across commercial, medical, market access, and other business functions. But the quality of every AI model, dashboard, and business insight depends on one thing: trusted data. As enterprise data grows across multiple platforms, pipelines, and business domains, maintaining consistent data quality has become increasingly difficult. Manual data quality rule creation, fragmented governance, and evolving business definitions make it challenging to scale data quality across the enterprise.
This post explores how Indegene built a production-grade, large language model (LLM) powered Data Quality Agent within its NEXT Unified Data Platform (NEXT UDP) on AWS. The agent generates context-aware data quality rules, validates them, and produces ready-to-run SQL and Python code under centralized, human-in-the-loop governance. Together, these capabilities help organizations improve data quality at enterprise scale while reducing the effort required to build and maintain data quality programs.
About Indegene
Indegene Limited (BSE: 544172, NSE: INDGN) is the strategic operating partner to the life sciences industry, orchestrating commercial, medical, R&D, and safety operations across the product lifecycle. Trusted by the world’s top 20 biopharma companies, Indegene helps run complex, highly regulated workflows at global scale. Indegene combines deep life sciences expertise with AI-embedded platforms and governed operating models to improve speed, consistency, and compliance across mission-critical operations. By integrating technology into how work is executed and scaled, Indegene helps life sciences organizations deliver measurable outcomes in regulated environments.
Why conventional data quality programs break down
Most organizations have established data quality processes but keeping them effective is becoming increasingly difficult as enterprise data ecosystems evolve.
One of the biggest challenges is the divide between technical and business rule ownership. Data engineers typically define technical rules that confirm whether data is structurally correct. For example, verifying that an HCP ID (Healthcare Professional ID) isn’t null, follows the required format, is unique, and maps to a valid reference record. Business teams define rules that determine whether the data makes sense in its operational context. For example, confirming that product market share totals 100 percent within a territory or that an HCP’s prescription activity aligns with their specialty and eligibility criteria. Because technical rules are often maintained in code while business rules reside in glossaries, SOPs, and subject-matter expertise, keeping them aligned and consistently governed across datasets and platforms becomes difficult.
Traditional validation techniques also have inherent limitations. Checks for null values, data types, uniqueness, or referential integrity remain essential, but they can’t capture business-specific context. Validating product hierarchies, customer segmentation, territory alignment, or commercial business rules requires domain knowledge that static validation logic can’t easily represent.
Creating and maintaining data quality rules is equally demanding. Every new dataset requires data profiling, rule definition, SQL development, validation, and ongoing maintenance. These rules are often spread across SQL scripts, pipeline code, spreadsheets, data catalogs, and tool-specific configurations. As schemas, business definitions, or validation thresholds change, the effort required to update and govern these rules grows significantly.
Additionally, the operational impact is significant. Onboarding new datasets becomes slower because many activities are repeated manually. Maintaining quality rules across hundreds of datasets consumes valuable engineering effort. Business users lose confidence when validation rules are difficult to understand, trace, or relate to business context.
As a result, data quality often becomes a periodic governance activity rather than a continuous, scalable capability.
Rethinking data quality with generative AI
Traditional data quality tools often operate as standalone capabilities, disconnected from the broader enterprise data ecosystem. As organizations scale AI and analytics, data quality needs to be embedded within the data platform itself, where governance, metadata, lineage, and analytics work together.
Indegene addresses this through the NEXT Unified Data Platform (NEXT UDP), a modern, AI-driven data platform built specifically for life sciences. NEXT UDP brings together fragmented commercial, medical, clinical, and enterprise data into a single, governed environment so organizations can manage the complete data lifecycle from ingestion to analytics. NEXT UDP provides several standalone AI agents, such as the Data Ingestion agent, Data Quality agent, Orchestration agent, Data Catalog agent, and SQL Bot Generator agent. You can use them standalone within your data pipelines or customize them further for agent-to-agent communication.
By integrating data ingestion, transformation, data quality, governance, metadata management, and analytics within a unified platform, NEXT UDP helps organizations:
- Consolidate fragmented enterprise data into a trusted source of truth.
- Standardize data quality and governance across business domains.
- Improve metadata visibility, lineage, and traceability.
- Reduce manual effort across data engineering workflows.
- Deliver trusted, AI-ready data for enterprise analytics and decision-making.
Indegene’s life sciences clients needed an agent that could run reliably in production and meet strict governance requirements, while giving the team flexibility in how they build it. Within NEXT UDP, Generative AI powers an intelligent Data Quality Agent that augments existing data quality processes. The agent boosts data engineer and data steward productivity by accelerating rule creation, simplifying validation, and helping maintain consistent governance at enterprise scale.
Indegene built the Data Quality Agent on a set of AWS services, with Amazon Bedrock, Strands Agents and Amazon Bedrock AgentCore at its core. Amazon Bedrock, provides a choice of foundation models (FMs), built on a foundation of security, privacy, and responsible AI controls. For life sciences organizations, this matters because their data often includes sensitive healthcare professional and patient information. Strands Agents streamlines building and updating the agent, and its evaluation tools help Indegene test rule accuracy, which is critical in regulated life sciences environments. The agent runs on Amazon Bedrock AgentCore, a platform to build, connect, and optimize agents at scale, with any framework or model. The solution also inherits existing AWS security and audit controls, so Indegene can focus on data quality logic instead of agent infrastructure.
What makes the Data Quality Agent unique? Two knowledge bases that power context-aware data quality
Any data quality tool can check for null values, duplicates, or invalid data types. Services such as AWS Glue Data Quality, a capability of AWS Glue, already recommend rules like these from data statistics, and they remain a strong foundation. The harder problem is determining whether the data makes sense for its business purpose. To address this, the Data Quality Agent uses two complementary knowledge bases that combine curated domain knowledge with experience gained from governed usage.
The first is an Internal Enterprise knowledge base: It combines internal business glossaries, data dictionaries, SOPs, regulatory guidance, approved data quality rules, validated scripts, profiling results, historical anomalies, and subject matter expert (SME) feedback. Content is ingested, vectorized, and retrieved through Retrieval Augmented Generation (RAG) to generate data quality rules grounded in enterprise standards and proven outcomes.
The second is an External Reference knowledge base: It’s built and periodically updated using approved standardized data quality frameworks, web search, and controlled web-scraping. When a dataset is selected, the agent analyzes table and column metadata and retrieves relevant data quality standards, reference values, and validation best practices from trusted sources. For example, for an NPI (National Provider Identifier) column, it can identify applicable quality standards, recommend format (the required 10-digit numeric format) and perform reference-validation checks.
How it works
The Data Quality Agent takes users from data profiling to rule creation and governance in a single workflow:
- Select source: Connect to a supported source such as Amazon Redshift or Snowflake.
- Select database and schema: Browse the available tables, with the rule count and active rule status for each.
- Profile data: The solution computes quality metrics and column patterns for the table, which users review.
- Detect anomalies: The solution flags anomalies against historical baselines, and users review the total, critical, and warning issues and the affected records.
- Manage rules: Review a table’s existing rules or start creating new ones.
- Choose rule mode: Pick Column Level, Table Level, or Custom Query.
- Generate rule suggestions: The agent uses metadata, profiling results, anomalies, existing rules, and knowledge base context to recommend technical and business rules, each with a severity.
- Review, refine and apply: Users edit or approve each recommendation.
- Generate and validate code: The agent produces SQL and Python for approved rules, and a validation step checks them.
- Store in the central rule repository: Approved rules become available for execution, monitoring, and audit.
Solution architecture of the generative AI-powered Data Quality Agent on AWS
The following diagram shows the end-to-end architecture of the Data Quality Agent on the NEXT UDP platform, from the data steward request to rule execution.
Data stewards and data quality analysts call the solution from the Indegene application. The following table provides a walkthrough of the solution pipeline and lists the AWS services used for each layer. It also captures other options the solution supports, so you can adapt it to your data volumes, existing tools, and operational preferences.
How the Data Quality Agent builds context
Before recommending any validation rule, the Data Quality Agent assembles context from six distinct sources, each answering a different question about the data:
Together, these sources let the agent reason like an experienced data steward. Grounding every recommendation in what the data is, what the standards require, how the enterprise defines quality, and what has already been implemented.
What the solution does
The Data Quality Agent generates several categories of checks and keeps data stewards in control of the results.
Anomaly checks generation
The solution generates anomaly checks that catch values which are technically valid but don’t match how a pharmaceutical market behaves. These include negative switch percentages, market shares that don’t add up to 100 percent within a market, inconsistent drug rankings (for example, a drug ranked first with a lower share than the drug ranked second), rating values outside the defined scale, and panel sizes that are too small or don’t match the number of respondents.
Context-aware rule generation with RAG
The Data Quality Agent generates rule recommendations in three modes:
The Create New Quality Rule interface provides dedicated Column Level, Table Level, and Custom Query modes.
The platform supports both technical validation and business-specific quality checks.
For technical rules, the agent
Auto-fetched and translated by AI Atlas. All rights belong to the original author.