人工智能
与 Crawl4AI 为 Amazon Bedrock 知识库准备网络内容
当构建 Retrieval 增强生成(RAG)应用程序时，回答质量取决于您摄入的数据。 Amazon贝德罗克知识基地是Amazon贝德罗克完全管理下的RAG能力。 使用 Amazon 贝德洛克管理的知识基地，您连接您的数据源，Amazon Bedrock 管理摄入，存储，索引和检索。 您不提供或缩放矢量存储，默认情况下嵌入并重新排序使用服务管理模式。
对于网页内容，从Amazon Bedrock Web Crawler数据源连接器开始，用于管理的知识库。 你给它种子URL或地点图授权你爬行，它爬行在你设定的范围和速率限制。 计划同步会随着页面变化而保持您的知识库流畅，连接器可以等待加载 JavaScript 的内容。 因此这个连接器是大多数公共网站内容的起点。
一些来源需要更多地控制知识库。 您可能想要只索引每个页面的主要内容区域，删除导航和其它锅炉板，或者将您自己的元数据属性附加到每个文档中。 对于该类源材料，你可以通过一个与Crawl4AI，开源 Python爬行器一起构建的准备步骤来扩展你的知识库的覆盖范围。 该步骤将每个页面的Markdown和一个元数据旁车写成Amazon简单存储服务(Amazon S3)，知识库通过它的Amazon S3数据源摄入。
在此文章中，我们演示您如何选择合适的网络摄入路径，并针对需要额外处理的内容执行自定义准备。 然后，您开始摄入一个管理下的知识库并验证检索质量。 配套的aws-样本存储器作为样本执行提供了完整的工作流程。
这是对已知的网页属性的批量摄入，而不是实时网页在推论时间的访问，例如Web Grounding with Amazon Nova.
解决方案概述
从Amazon Bedrock Web Crawler开始，当它能够到达并索引你所需要的页面时。 它尊重机器人.txt，在第一次完全爬行后，每次同步捕获只添加，更新和删除内容。
当公共来源材料在摄入前需要一个或多个下列内容时，使用 Crawl4AI 准备路径：
- 结构化的Markdown准备：在保留标题、列表和表格的同时，使用带有穿插内容过滤器的默认MarkdownGenerator以减少导航和其他锅炉板。 因为挤压是高压的，审查你们网站的代表性产出。
- 定点取出：只提取一个可选 CSS 选择器匹配的页面区域。 按种子来源、页面/路径/主机范围、深度和页盖以及URL模式来限制宽度第一。
- 自定义元数据：附加源 URL, craw timetamp，内容类型，以及您自己对每个文档的属性，这样您就可以过滤它们上的检索结果。
由于管理和客户管理的知识库都支持Amazon S3数据源，因此Crawl4AI路径与任意一种类型一起工作。 这个员额使用一个管理下的知识库和一个客户管理的知识库不同的指出。
此帖余随伴样。 将它的架构和默认作为起点，而不是Amazon贝德罗克知识库的要求。
默认情况下，样本在一个公共子网中运行AWS Fargate任务，通过互联网网关到达您配置的网站。 Crawl4AI的作用在它准备了源文件和元数据辅助车后即告结束。 从摄入开始，Amazon Bedrock知识库处理解析，块化，嵌入，索引化和检索。 以下的图显示 Crawl4AI 路径从调度到检索。
图表图中的编号步骤到以下工作流程：
- Amazon EventBridge在所配置的日程安排中开始Amazon弹性集装箱服务(Amazon ECS)任务。
- 任务爬上您配置的公开网站，并为每个页面准备一个元数据旁卡。
- 任务将准备的文件上传到支持知识库数据源的Amazon S3桶。
- 任务开始Amazon Bedrock知识库摄入工作。
- Amazon Bedrock 知识库读取新的和已改变的对象，分析它们，并应用你选择的块策略。
- 它在知识库的矢量储存库中建立嵌入和索引，Amazon Bedrock为管理的知识库管理。
- 您的基因AI应用程序询问相关块的知识库，或以引用方式生成的答复。
任务从启动时的AWS系统管理器参数存储器读取其非秘密的管道配置。
工作流程只处理公共页面，而不执行经认证的会议。 对于需要登录的站点，首先评价Amazon Bedrock Web Crawler认证选项，包括基本，基于形式，和SAML认证。 以认证方式扩展样本需要会话处理，AWS保密管理器中的证书，以及安全审查。
工作流程如何运作
AWS Cloud Development Kit(AWS CDK)应用软件在部署时将执行包装在一个容器图像中。 当时刻表起火时，容器会运行脚本/容器 enterpoint.py并按顺序执行爬行，上传和摄入。 以下三个步骤解释了核心逻辑。
步骤1：爬行和提取内容
爬行步骤执行边框爬行，并为每个成功准备的页面写一个带有元数据侧车的Markdown文件。 以下节录简化。 模块src/crawl4ai bedrock kb/crawl.py包含完整的种子间隔，URL过滤和范围，页数限制，以及方向检查：
Amazon Bedrock Knowledge Bases支持Amazon S3数据源的几种文档格式，包括纯文本，HTML,Markdown,PDF，和DOCX. 工作流程写入 Markdown 是因为它保存标题，列表和表格，而不包含所有原始的 HTML 输入。
步骤2：上传到 Amazon S3
上传步骤会在工作流程的 s3Prefix 下，将每个已准备好的Markdown 文档及其匹配的元数据侧车发送到配置的 S3 桶中。 对于page.md，侧车必须命名为page.md.metadata.json，完整的源文件名带有.metadata.json附件。 工作流程将每台侧车存放在与其文档相同的S3位置。 Amazon Bedrock将每个元数据侧车限制在10KB. 如果侧车超过这个限制，运行会在其写出任何文件之前出现错误。
副车厢中的每个属性都使用管理及客户管理的知识库接受Amazon S3数据源的打字格式。 以下简称侧车，格式化为可读性，显示 3 工作流程所撰写的属性：
管道只上传当前爬行产生的路径，因此不发送呆滞的本地文件。
第3步：让Amazon Bedrock知识库块、嵌入和索引
在内容在S3之后，一个摄入工作将每个文件交给数据源的摄入管道。 默认情况下，一个管理的知识库会用它所管理的解析器来解析每个文件，它会自动为您的内容选择解析方法。 连接数据源时选择块策略。 在创建生产数据源之前，管理下的知识库提供以下分散的战略：
- 默认块使用300个令牌的固定大小块，重叠20%。
- 固定大小块分割内容，以您设定的最大符号计数和重叠百分比。
- 对于您已经拆分的内容，没有将每个文件当作一个块处理。
春克边界和检索结果取决于内容和所选策略，因此将选项与代表性页面和评价问题进行比较。 为了日后改变策略，创建一个新的数据源。 更多信息见自定义摄入数据源。 由客户管理的知识库也提供语义学和分级划分和附加解析选项。 详情见内容如何对知识库进行分割。
工作流程随后开始一个摄入工作，因此知识库处理新的文件。 以下简化节录来自src/crawl4ai bedrock kb/ingest.py 显示的调用：
同样的"Start IntroductionJob"和"Get IntroductionJob"(Get IntroductionJob)要求两种知识库类型的工作，也可以在控制台上监控进度。
部署解决方案
在本节中，您将工作流程与 AWS CDK 一起部署到您的 AWS 账户中。 只有知识库的先决条件在两种知识库类型之间有所不同。 完成后，请跟随清理部分的步骤移除资源。
先决条件
在开始前，请确保你有：
- AWS账户。
- 下列知识库之一：
- 该员额在AWS区域使用管理的知识库，那里有管理的知识库。 有关指令，请参见"创建管理的知识库"和"Amazon S3"连接器。 在数据源上，将元数据文件前缀留为空，因为工作流程会将每个元数据侧刻符与文档一起写入。
- 现有客户管理的知识库。 要添加 Amazon S3 数据源，请参见 Connect to Amazon S3 您的知识库。 工作流程为每个文档写一个元数据旁卡，以确认您的矢量存储可以存储和过滤这些属性，如元数据和过滤中描述的那样。 例如，Amazon OpenSearch 服务器无向量索引必须使用faiss引擎。 Amazon Aurora矢量指数需要自定义元数据列或每个元数据属性的列。 用 Amazon S3 矢量器，能力为 Amazon S3，您可以在每个矢量中附加最多 1 KB 的自定义元数据，所以保持您添加的属性短。
- 该知识库中的Amazon S3数据来源，其状况为AVAILable。 它可以是新的数据源，也可以是您已经使用的数据源。 它必须使用您为此工作流程配置的 S3 桶。 如果数据源通过前缀或文件模式过滤内容，过滤器必须允许工作流程在其s3Prefix(默认web-content/)下写入的文件。
- 知识库，S3桶，还有你部署的堆积物。
- 授权浏览您配置的公共网站。
- 部署工作站，配备Git,Python 3.10或以后，Node.js,AWS指挥线接口(AWS CLI)版本2,AWS CDK CLI v2，以及运行的Docker兼容容器引擎。 部署指南列出了最小的AWS CDK CLI版本。
- AWS证书配置在部署工作站，并允许拖曳目标环境和部署CDK堆栈。
- 目标账户和地区中一个带靴子的CDK环境。 部署步骤包括靴子陷阱命令。
你只需要工作站来建造和部署部署后，时刻表在AWS运行任务。
部署预定的工作流程
清除同伴寄存器并安装部署依赖性：
保持Docker运行。 将 AWS DEFAULT Region 设定为您的知识库区域，因为 cdk 部署使用来自您的环境或 AWS 配置，而不是靴子命令中的区域。 必要时，将目标账户和区域绑一次。 然后用您的知识库、数据源、 S3 桶和授权种子 URL 来部署堆栈：
替换每个示例值，包括 AcoUNT ID，区域，以及所需的四个上下文密钥，替换为您自己的。 示例值在策略上是有效的，因此一个未经编辑的命令会部署一个堆栈，以指向您不拥有的桶。 也可以将这些密钥设置在cdk.json中。
最大深度和最大Pages值将第一次爬行限制在一个链接级别和25页。 在您审查已准备好的输出后提升它们。 要预览哪个页面
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI
When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base, you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default.
For web content, start with the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. You give it the seed URLs or sitemaps that you’re authorized to crawl, and it crawls within the scope and rate limits that you set. Scheduled syncs keep your knowledge base current as pages change, and the connector can wait for content that loads with JavaScript. There’s no crawler for you to build, host, or operate, so this connector is the starting point for most public website content.
Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the reach of your knowledge base with a preparation step built with Crawl4AI, an open source Python crawler. The step writes Markdown and a metadata sidecar for each page to Amazon Simple Storage Service (Amazon S3), and the knowledge base ingests both through its Amazon S3 data source.
In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion aws-samples repository provides the complete workflow as a sample implementation.
This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.
Solution overview
Start with the Amazon Bedrock Web Crawler when it can reach and index the pages you need. It respects robots.txt, and after the first full crawl, each sync captures only added, updated, and deleted content.
Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:
- Structured Markdown preparation: Use DefaultMarkdownGenerator with a pruning content filter to reduce navigation and other boilerplate while preserving headings, lists, and tables. Because pruning is heuristic, review representative output from your sites.
- Bounded, targeted extraction: Extract only the page region that an optional CSS selector matches. Constrain breadth-first traversal by seed origin, page/path/host scope, depth and page caps, and URL patterns.
- Custom metadata: Attach source URLs, crawl timestamps, content type, and your own attributes to each document, so that you can filter retrieval results on them.
Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.
The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.
By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval. The following diagram shows the Crawl4AI path from the schedule through to retrieval.
The numbered steps in the diagram map to the following workflow:
- Amazon EventBridge starts the Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate on the configured schedule.
- The task crawls the public websites you configured and prepares Markdown with a metadata sidecar for each page.
- The task uploads the prepared files to the Amazon S3 bucket that backs the knowledge base data source.
- The task starts an Amazon Bedrock Knowledge Bases ingestion job.
- Amazon Bedrock Knowledge Bases reads the new and changed objects, parses them, and applies the chunking strategy you selected.
- It creates embeddings and indexes them in the knowledge base’s vector store, which Amazon Bedrock manages for a managed knowledge base.
- Your generative AI application queries the knowledge base for relevant chunks or for a generated response with citations.
The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup.
The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the Amazon Bedrock Web Crawler authentication options first, which include basic, form-based, and SAML authentication. Extending the sample with authentication requires session handling, credentials in AWS Secrets Manager, and a security review.
How the workflow works
The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs scripts/container_entrypoint.py and executes crawl, upload, and ingestion in sequence. The following three steps explain the core logic.
Step 1: Crawl and extract content
The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module src/crawl4ai_bedrock_kb/crawl.py contains the complete seed pacing, URL filtering and scoping, page limits, and redirect checks:
Amazon Bedrock Knowledge Bases supports several document formats for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow writes Markdown because it preserves headings, lists, and tables without carrying all raw HTML into ingestion.
Step 2: Upload to Amazon S3
The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s s3Prefix. For page.md, the sidecar must be named page.md.metadata.json, the complete source filename with .metadata.json appended. The workflow stores each sidecar in the same S3 location as its document. Amazon Bedrock limits each metadata sidecar to 10 KB. If a sidecar would exceed that limit, the run stops with an error before it writes either file.
Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:
The pipeline uploads only the paths produced by the current crawl, so stale local files are not sent.
Step 3: Let Amazon Bedrock Knowledge Bases chunk, embed, and index
After content is in S3, an ingestion job hands each document to the data source’s ingestion pipeline. By default, a managed knowledge base parses each file with its managed parser, which automatically selects a parsing approach for your content. You select the chunking strategy when you connect the data source. You can’t change it afterward, so evaluate the options before you create a production data source. A managed knowledge base offers the following chunking strategies:
- Default chunking uses fixed-size chunks of 300 tokens with 20 percent overlap.
- Fixed-size chunking splits content by a maximum token count and an overlap percentage that you set.
- No chunking treats each file as a single chunk, for content that you have already split.
Chunk boundaries and retrieval results depend on the content and the selected strategy, so compare the options with representative pages and evaluation questions. To change strategies later, create a new data source. For more information, see Customize ingestion for a data source. A customer-managed knowledge base also offers semantic and hierarchical chunking and additional parsing options. For details, see How content chunking works for knowledge bases.
The workflow then starts an ingestion job so the knowledge base processes the new documents. The following simplified excerpt from src/crawl4ai_bedrock_kb/ingest.py shows the call:
The same StartIngestionJob and GetIngestionJob calls work for both knowledge base types, and you can also monitor progress on the console.
Deploy the solution
In this section, you deploy the workflow into your AWS account with AWS CDK. Only the knowledge base prerequisites differ between the two knowledge base types. When you’re finished, follow the steps in the Clean up section to remove the resources.
Prerequisites
Before you start, make sure that you have:
- An AWS account.
- One of the following knowledge bases:
- A managed knowledge base, which this post uses, in an AWS Region where managed knowledge bases are available. For instructions, see Create a managed knowledge base and the Amazon S3 connector. On the data source, leave Metadata files prefix empty, because the workflow writes each metadata sidecar alongside its document.
- An existing customer-managed knowledge base. To add an Amazon S3 data source, see Connect to Amazon S3 for your knowledge base. The workflow writes a metadata sidecar for every document, so confirm that your vector store can store and filter those attributes, as described in Metadata and filtering. For example, an Amazon OpenSearch Serverless vector index must use the faiss engine. An Amazon Aurora vector index needs a custom metadata column or a column for each metadata attribute. With Amazon S3 Vectors, a capability of Amazon S3, you can attach up to 1 KB of custom metadata to each vector, so keep attributes that you add short.
- An Amazon S3 data source in that knowledge base whose status is AVAILABLE. It can be a new data source or one you already use. It must use the S3 bucket that you configure for this workflow. If the data source filters content by prefix or file pattern, the filters must allow the files that the workflow writes under its s3Prefix (default web-content/).
- The knowledge base, the S3 bucket, and the stack you deploy all in one Region.
- Authorization to crawl the public websites you configure.
- A deployment workstation with Git, Python 3.10 or later, Node.js, the AWS Command Line Interface (AWS CLI) version 2, the AWS CDK CLI v2, and a running Docker-compatible container engine. The deployment guide lists the minimum AWS CDK CLI version.
- AWS credentials configured on the deployment workstation with permission to bootstrap the target environment and deploy the CDK stack.
- A bootstrapped CDK environment in the target account and Region. The deployment steps include the bootstrap command.
You need the workstation only to build and deploy. After deployment, the schedule runs the task in AWS.
Deploy the scheduled workflow
Clone the companion repository and install the deployment dependencies:
Keep Docker running. Set AWS_DEFAULT_REGION to your knowledge base’s Region first, because cdk deploy uses the Region from your environment or AWS profile, not the one in the bootstrap command. Bootstrap the target account and Region once, if needed. Then deploy the stack with your knowledge base, data source, S3 bucket, and authorized seed URLs:
Replace every example value, including ACCOUNT_ID, REGION, and the four required context keys, with your own. The example values are syntactically valid, so an unedited command deploys a stack that points at a bucket you don’t own. You can also set these keys in cdk.json.
The maxDepth and maxPages values limit the first crawl to one link level and 25 pages. Raise them after you review the prepared output. To preview which pages
Auto-fetched and translated by AI Atlas. All rights belong to the original author.