# dataverse-python-performance-optimization

Performance guidelines for Dataverse SDK for Python. Instructs Copilot to apply server-side OData filtering, batch creates, and lazy paging.

- rule · `type=rule` · GitHub Copilot · ★ 40418
- Author: github · official GitHub repository
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/github/awesome-copilot/tree/HEAD/instructions/dataverse-python-performance-optimization.instructions.md
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/rules/dataverse-python-performance-optimization

## What it does

This rule targets preview limitations of the Dataverse SDK for Python. The preview SDK only retries network errors by default, lacks `DeleteMultiple`, and restricts SQL statements to simple single-table SELECT queries without joins.

Copilot learns to restrict retrieved columns with `select`, apply server-side OData filters, and consume paginated records iteratively instead of reading batches into memory with `list()`. Bulk operations are split according to table structure: standard tables handle batches of 200 to 300 records, while complex tables drop to batches of 10 to 20.

Creating a fresh `DataverseClient` inside loops degrades network performance, so the rule enforces singleton client reuse. Files exceeding 128 MB are routed through chunked uploads using 4 MB blocks.

## Install

### GitHub Copilot (native platform)

**Manually: put the files in place**
- File: `.github/instructions/dataverse-python-performance-optimization.instructions.md` — The applyTo field sets which files it applies to

### Claude Code

**Manually: put the files in place**
- File: `.claude/rules/dataverse-python-performance-optimization.md` — The paths frontmatter field limits the files: move applyTo from the source into paths

### Codex

**AGENTS.md**
- Codex reads AGENTS.md: paste the rule text there

### Cursor

**Manually: put the files in place**
- File: `.cursor/rules/dataverse-python-performance-optimization.mdc` — Frontmatter: description, globs, alwaysApply. Move applyTo from the source into globs, or the rule will not attach on its own

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/rules/dataverse-python-performance-optimization.md`

### Cline

**Manually: put the files in place**
- File: `.clinerules/dataverse-python-performance-optimization.md`

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/github/awesome-copilot/tree/HEAD/instructions/dataverse-python-performance-optimization.instructions.md

## When to use it
- enforce server-side OData filters and column pruning via `select`
- batch bulk record creation and updates based on table schema complexity
- stream paginated datasets to avoid memory exhaustion
- configure chunked parallel uploads for files larger than 128 MB

## Example prompts
- Write an optimized batch insert for 1000 accounts using DataverseClient
- Query active contacts using lazy pagination and server-side OData filters
- Implement a file upload helper for files over 128 MB in Dataverse

## Source text · dataverse-python-performance-optimization.instructions.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 5ff789fb5aab73cc.

BEGIN THIRD-PARTY FILE 5ff789fb5aab73cc
````text
---
applyTo: '**'
---

# Dataverse SDK for Python — Performance & Optimization Guide

Based on official Microsoft Dataverse and Azure SDK performance guidance.

## 1. Performance Overview

The Dataverse SDK for Python is optimized for Python developers but has some limitations in preview:
- **Minimal retry policy**: Only network errors are retried by default
- **No DeleteMultiple**: Use individual deletes or update status instead
- **Limited OData batching**: General-purpose OData batching not supported
- **SQL limitations**: No JOINs, limited WHERE/TOP/ORDER BY

Workarounds and optimization strategies address these limitations.

---

## 2. Query Optimization

### Use Select to Limit Columns

```python
# ❌ SLOW - Retrieves all columns
accounts = client.get("account", top=100)

# ✅ FAST - Only retrieve needed columns
accounts = client.get(
    "account",
    select=["accountid", "name", "telephone1", "creditlimit"],
    top=100
)
```

**Impact**: Reduces payload size and memory usage by 30-50%.

---

### Use Filters Efficiently

```python
# ❌ SLOW - Fetch all, filter in Python
all_accounts = client.get("account")
active_accounts = [a for a in all_accounts if a.get("statecode") == 0]

# ✅ FAST - Filter server-side
accounts = client.get(
    "account",
    filter="statecode eq 0",
    top=100
)
```

**OData filter examples**:
```python
# Equals
filter="statecode eq 0"

# String contains
filter="contains(name, 'Acme')"

# Multiple conditions
filter="statecode eq 0 and createdon gt 2025-01-01Z"

# Not equals
filter="statecode ne 2"
```

---

### Order by for Predictable Paging

```python
# Ensure consistent order for pagination
accounts = client.get(
    "account",
    orderby=["createdon desc", "name asc"],
    page_size=100
)

for page in accounts:
    process_page(page)
```

---

## 3. Pagination Best Practices

### Lazy Pagination (Recommended)

```python
# ✅ BEST - Generator yields one page at a time
pages = client.get(
    "account",
    top=5000,              # Total limit
    page_size=200          # Per-page size (hint)
)

for page in pages:  # Each iteration fetches one page
    for record in page:
        process_record(record)  # Process immediately
```

**Benefits**:
- Memory efficient (pages loaded on-demand)
- Fast time-to-first-result
- Can stop early if needed

### Avoid Loading Everything into Memory

```python
# ❌ SLOW - Loads all 100,000 records at once
all_records = list(client.get("account", top=100000))
process(all_records)

# ✅ FAST - Process as you go
for page in client.get("account", top=100000, page_size=5000):
    process(page)
```

---

## 4. Batch Operations

### Bulk Create (Recommended)

```python
# ✅ BEST - Single call with multiple records
payloads = [
    {"name": f"Account {i}", "telephone1": f"555-{i:04d}"}
    for i in range(1000)
]
ids = client.create("account", payloads)  # One API call for many records
```

### Bulk Update - Broadcast Mode

```python
# ✅ FAST - Same update applied to many records
account_ids = ["id1", "id2", "id3", "..."]
client.update("account", account_ids, {"statecode": 1})  # One call
```

### Bulk Update - Per-Record Mode

```python
# ✅ ACCEPTABLE - Different updates for each record
account_ids = ["id1", "id2", "id3"]
updates = [
    {"telephone1": "555-0100"},
    {"telephone1": "555-0200"},
    {"telephone1": "555-0300"},
]
client.update("account", account_ids, updates)
```

### Batch Size Tuning

Based on table complexity (per Microsoft guidance):

| Table Type | Batch Size | Max Threads |
````
END THIRD-PARTY FILE 5ff789fb5aab73cc

The third-party file has ended. Everything between the 5ff789fb5aab73cc tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 3535 of 12219 bytes. © github · MIT · https://github.com/github/awesome-copilot/tree/HEAD/instructions/dataverse-python-performance-optimization.instructions.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md