Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.llms_txt_reader import LLMsTxtReader
from agno.vectordb.pgvector import PgVector
db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"
knowledge = Knowledge(
name="LLMs.txt Docs",
vector_db=PgVector(table_name="llms_txt_docs", db_url=db_url),
)
knowledge.insert(
url="https://docs.agno.com/llms.txt",
reader=LLMsTxtReader(max_urls=10),
)
agent = Agent(
knowledge=knowledge,
search_knowledge=True,
)
agent.print_response("What is Agno?", markdown=True)
Set up your virtual environment
uv venv --python 3.12
source .venv/bin/activate
uv venv --python 3.12
.venv\Scripts\activate
Install dependencies
uv pip install -U beautifulsoup4 sqlalchemy psycopg pgvector agno openai
Set environment variables
export OPENAI_API_KEY=xxx
Run PgVector
docker run -d \
-e POSTGRES_DB=ai \
-e POSTGRES_USER=ai \
-e POSTGRES_PASSWORD=ai \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v pgvolume:/var/lib/postgresql/data \
-p 5532:5432 \
--name pgvector \
agno/pgvector:16
Run Agent
python examples/basics/knowledge/concepts/readers/overview/llms_txt_reader.py
| Parameter | Type | Default | Description |
|---|---|---|---|
url | str | Required | URL of the llms.txt file to read |
max_urls | int | 20 | Maximum number of linked URLs to fetch from the file |
timeout | int | 60 | HTTP timeout in seconds |
proxy | Optional[str] | None | Optional HTTP proxy URL |
skip_optional | bool | False | Skip entries under the ## Optional section |
chunking_strategy | Optional[ChunkingStrategy] | FixedSizeChunking() | Strategy for chunking content |
allowed_hosts | Optional[List[str]] | None | Hostnames the reader is allowed to fetch from. See Restricting URL Fetches. |
Was this page helpful?