ida-pro-mcp/devdocs/test-framework.md

IDA Pro MCP Test Framework

This document provides detailed documentation for the test framework used in ida-pro-mcp.

Overview

The test framework is a custom, lightweight testing solution designed specifically for testing IDA Pro MCP tools. It supports:

• Inline tests - Tests defined right after the functions they test in api_*.py files
• IDA console runner - Run tests interactively inside IDA Pro
• Standalone runner - Run tests headlessly using idalib
• Code coverage - Measure code coverage of API implementations
• Binary-agnostic assertions - Tests that work on any binary

Architecture

src/ida_pro_mcp/
├── ida_mcp/
│   ├── framework.py          # Test framework core
│   │   ├── @test decorator   # Register test functions
│   │   ├── run_tests()       # Test runner
│   │   ├── TestResults       # Result aggregation
│   │   ├── assert_*          # Assertion helpers
│   │   └── get_any_*         # Test data helpers
│   ├── tests/                # Separate test files
│   │   ├── test_api_core.py
│   │   ├── test_api_analysis.py
│   │   ├── test_api_memory.py
│   │   ├── test_api_modify.py
│   │   ├── test_api_types.py
│   │   ├── test_api_stack.py
│   │   └── test_api_resources.py
│   ├── api_core.py           # API implementations
│   ├── api_analysis.py
│   └── ...
├── test.py                   # Standalone runner (idalib)
└── ...
tests/
├── crackme03.elf             # Test binary
└── binary_info.py            # Binary info extraction

Test Registration

The @test Decorator

from .tests import test

@test()
def test_idb_meta():
    """Test description"""
    meta = idb_meta()
    assert_has_keys(meta, "path", "module", "base")

Parameters

Parameter	Type	Default	Description
binary	str	""	Specific binary this test runs for
skip	bool	False	Skip this test

Examples

# Basic test
@test()
def test_basic():
    pass

# Skipped test
@test(skip=True)
def test_broken():
    pass

Test Placement

Tests should be placed immediately after the function they test:

@tool
@idasync
def idb_meta() -> Metadata:
    """Get IDB metadata"""
    return Metadata(...)


@test()
def test_idb_meta():
    """idb_meta returns valid metadata"""
    meta = idb_meta()
    assert_has_keys(meta, "path", "module")

This keeps tests close to implementation and makes it easy to see test coverage.

Test Auto-Discovery

Tests are automatically discovered and categorized by their module:

# In api_core.py
@test()
def test_idb_meta():  # Registered as category="api_core"
    pass

# In api_analysis.py
@test()
def test_decompile():  # Registered as category="api_analysis"
    pass

Assertion Helpers

Import helpers from tests.py:

from .tests import (
    test,
    assert_has_keys,
    assert_valid_address,
    assert_non_empty,
    assert_is_list,
    assert_all_have_keys,
)

assert_has_keys(d, *keys)

Assert a dict contains all specified keys:

meta = idb_meta()
assert_has_keys(meta, "path", "module", "base", "size")

assert_valid_address(addr)

Assert a string is a valid hex address:

assert_valid_address("0x401000")    # OK
assert_valid_address("-0x10")       # OK (negative)
assert_valid_address("401000")      # FAIL - no 0x prefix
assert_valid_address("not_addr")    # FAIL - not hex

assert_non_empty(value)

Assert value is not None and not empty:

assert_non_empty("hello")           # OK
assert_non_empty([1, 2, 3])         # OK
assert_non_empty("")                # FAIL
assert_non_empty([])                # FAIL
assert_non_empty(None)              # FAIL

assert_is_list(value, min_length=0)

Assert value is a list with minimum length:

assert_is_list([1, 2, 3])           # OK
assert_is_list([1, 2], min_length=2) # OK
assert_is_list([1], min_length=2)   # FAIL - too short
assert_is_list("string")            # FAIL - not a list

assert_all_have_keys(items, *keys)

Assert all dicts in a list have specified keys:

funcs = [{"addr": "0x1", "name": "a"}, {"addr": "0x2", "name": "b"}]
assert_all_have_keys(funcs, "addr", "name")  # OK
assert_all_have_keys(funcs, "addr", "size")  # FAIL - missing "size"

Test Data Helpers

get_any_function() -> Optional[str]

Get address of first function (for tests needing a valid function):

fn_addr = get_any_function()
if not fn_addr:
    return  # Skip test if no functions

result = decompile(fn_addr)
assert result[0]["code"] is not None

get_any_string() -> Optional[str]

Get address of first string:

str_addr = get_any_string()
if not str_addr:
    return  # Skip if no strings

result = get_string(str_addr)
assert result[0]["value"] is not None

get_first_segment() -> Optional[tuple[str, str]]

Get first segment's (start, end) addresses:

seg = get_first_segment()
if not seg:
    return

start_addr, end_addr = seg
result = get_bytes({"addr": start_addr, "size": 16})

Running Tests

From IDA Console

from ida_mcp.tests import run_tests

# Run all tests
run_tests()

# Run specific category
run_tests(category="api_core")

# Run tests matching pattern
run_tests(pattern="*decompile*")

# Combine filters
run_tests(pattern="test_list_*", category="api_core")

# Stop on first failure
run_tests(stop_on_failure=True)

# Quiet mode (just summary)
run_tests(verbose=False)

From Command Line

# Basic usage
ida-mcp-test tests/crackme03.elf

# Filter by category
ida-mcp-test tests/crackme03.elf --category api_core

# Filter by pattern
ida-mcp-test tests/crackme03.elf --pattern "*meta*"

# List available tests
ida-mcp-test tests/crackme03.elf --list

# Stop on first failure
ida-mcp-test tests/crackme03.elf --stop-on-failure

# With code coverage
ida-mcp-test tests/crackme03.elf --coverage

# HTML coverage report
ida-mcp-test tests/crackme03.elf --coverage --coverage-report html

Code Coverage

Code coverage is measured using the coverage package with uv run coverage run.

Running with Coverage

# Run tests with coverage
uv run coverage run -m ida_pro_mcp.test crackme03.elf

# Show coverage report in terminal
uv run coverage report --show-missing

# Generate HTML report
uv run coverage html
open htmlcov/index.html

# Generate XML report (for CI)
uv run coverage xml

# Generate JSON report
uv run coverage json

# Combine multiple coverage runs
uv run coverage combine
uv run coverage report

Coverage Configuration

Create a .coveragerc file or add to pyproject.toml:

[tool.coverage.run]
source = ["ida_pro_mcp.ida_mcp"]
omit = [
    "*/tests.py",
    "*/zeromcp/*",
    "*/__init__.py",
]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
]

Test Patterns

1. Schema Validation

Verify return types match expected structure:

@test()
def test_idb_meta():
    """idb_meta returns properly structured metadata"""
    meta = idb_meta()
    assert_has_keys(meta, "path", "module", "base", "size", "md5", "sha256")
    assert_valid_address(meta["base"])

2. Non-Empty Results

Verify APIs return data on valid binaries:

@test()
def test_list_funcs():
    """list_funcs returns at least one function"""
    result = list_funcs({})
    assert_is_list(result, min_length=1)
    assert_is_list(result[0]["data"], min_length=1)

3. Error Handling

Verify invalid inputs raise IDAError:

from .sync import IDAError

@test()
def test_decompile_invalid():
    """decompile raises IDAError for invalid address"""
    try:
        decompile("0xDEADBEEFDEADBEEF")
        assert False, "Expected IDAError"
    except IDAError:
        pass  # Expected

4. Referential Integrity

Verify cross-API consistency:

@test()
def test_functions_are_decompilable():
    """Functions from list_funcs can be decompiled"""
    result = list_funcs({"count": 3})
    for fn in result[0]["data"]:
        dec = decompile(fn["addr"])
        # Should either have code or specific error, not crash
        assert "code" in dec[0] or "error" in dec[0]

5. Round-Trip with Cleanup

Test modifications and restore original state:

@test()
def test_rename_roundtrip():
    """rename function works and can be undone"""
    fn_addr = get_any_function()
    if not fn_addr:
        return

    # Get original name
    original = lookup_funcs(fn_addr)[0]["fn"]["name"]

    try:
        # Rename
        rename({"func": [{"addr": fn_addr, "name": "__test__"}]})

        # Verify
        new_name = lookup_funcs(fn_addr)[0]["fn"]["name"]
        assert new_name == "__test__"
    finally:
        # Restore
        rename({"func": [{"addr": fn_addr, "name": original}]})

6. Batch Operations

Verify batch input handling:

@test()
def test_batch_operations():
    """Operations handle multiple inputs"""
    result = list_funcs({"count": 3})
    addrs = [fn["addr"] for fn in result[0]["data"]]

    # Batch lookup
    results = lookup_funcs(addrs)
    assert len(results) == len(addrs)

Test Results

TestResult Structure

@dataclass
class TestResult:
    name: str                          # Test function name
    category: str                      # Module category (api_core, etc.)
    status: Literal["passed", "failed", "skipped"]
    duration: float                    # Execution time in seconds
    error: Optional[str]               # Error message if failed
    traceback: Optional[str]           # Full traceback if failed

TestResults Aggregate

@dataclass
class TestResults:
    passed: int
    failed: int
    skipped: int
    total_time: float
    results: list[TestResult]

Programmatic Access

results = run_tests(verbose=False)

print(f"Passed: {results.passed}")
print(f"Failed: {results.failed}")

for r in results.results:
    if r.status == "failed":
        print(f"{r.name}: {r.error}")
        print(r.traceback)

Adding New Tests

Step-by-Step

1. Identify the function to test in the appropriate api_*.py file

2. Add imports at the top of the file:

   from .tests import (
       test,
       assert_has_keys,
       assert_valid_address,
       # ... other helpers you need
   )
   

3. Write the test immediately after the function:

   @tool
   @idasync
   def my_function(...):
       ...
   
   
   @test()
   def test_my_function():
       """Description of what the test verifies"""
       result = my_function(...)
       assert_has_keys(result, "key1", "key2")
   

4. Run the test:

   ida-mcp-test tests/crackme03.elf --pattern "test_my_function"
   

Best Practices

1. Test name: Use test_<function_name> or test_<function_name>_<scenario>

2. Docstring: First line describes what the test verifies

3. Binary-agnostic: Don't hardcode addresses or expected values

4. Skip gracefully: Return early if required data isn't available

   fn_addr = get_any_function()
   if not fn_addr:
       return  # Skip, don't fail
   

5. Cleanup: Restore state for tests that modify the IDB

   try:
       # Modify
   finally:
       # Restore
   

6. Error handling: Use try/except IDAError for expected errors

Coverage-Guided Test Development

Tests should be coverage-guided: use coverage data to identify gaps in our wrapper code, then write tests that exercise those specific code paths.

Methodology

Key Principle: We're testing our wrapper code, not IDA's correctness. Don't write tests that verify IDA returns expected values - IDA works. Instead, write tests that trigger edge cases and error paths in our code.

Workflow

1. Run coverage:

   uv run coverage run -m ida_pro_mcp.test tests/crackme03.elf
   uv run coverage report --show-missing --include="src/ida_pro_mcp/ida_mcp/api_*.py"
   

2. Identify uncovered lines and understand what conditions trigger them

3. Determine test type:

   • Generic test: If the gap can be triggered with invalid/edge-case inputs (preferred)
   • Binary-specific test: Only if the gap requires specific binary properties

4. Write minimal test that triggers the specific code path

5. Re-run coverage to verify the gap is closed

Example: Coverage-Guided Test Development

Coverage shows lines 252-256 in api_core.py are uncovered:

except ValueError:
    results.append(
        {"input": text, "result": None, "error": f"Invalid number: {text}"}
    )

This is an error path for invalid input. Write a generic test:

@test()
def test_int_convert_invalid_text():
    """int_convert handles invalid number text (covers lines 252-256)"""
    result = int_convert({"text": "not_a_number"})
    assert result[0]["result"] is None
    assert "Invalid number" in result[0]["error"]

Deterministic Sampling

For broader test coverage, test on multiple items rather than just one. The framework provides helpers that select a deterministic sample based on binary name.

Sample Size Configuration

Sample size is configured via command line:

# Default: 5 items
uv run ida-mcp-test tests/crackme03.elf

# More thorough testing with 10 items
uv run ida-mcp-test tests/crackme03.elf --sample-size 10

# Quick smoke test with 2 items
uv run ida-mcp-test tests/crackme03.elf --sample-size 2

Sampling Helpers

from .tests import get_n_functions, get_n_strings, get_functions_with_calls

@test()
def test_decompile_multiple():
    """decompile works on multiple functions (sampling test)"""
    addrs = get_n_functions()  # Uses configured sample size
    if len(addrs) < 2:
        return

    results = decompile(addrs)
    assert len(results) == len(addrs)
    # At least some should succeed
    successes = [r for r in results if r.get("code")]
    assert len(successes) > 0

Available Helpers

Helper	Description
get_n_functions(n=None)	N deterministically-selected function addresses
get_n_strings(n=None)	N deterministically-selected string addresses
get_functions_with_calls()	Functions that contain call instructions
get_functions_with_callers()	Functions that are called by others
get_data_address()	Address in a data (non-code) segment
get_unmapped_address()	Invalid address for error path testing

Test Categories

1. Schema Tests

Verify return structure matches expected schema:

@test()
def test_idb_meta_schema():
    result = idb_meta()
    assert_has_keys(result, "path", "module", "base", "size")

2. Error Path Tests

Verify error handling for invalid inputs:

@test()
def test_decompile_invalid_address():
    """Covers error handling path"""
    result = decompile(get_unmapped_address())
    assert result[0].get("error") is not None

3. Edge Case Tests

Cover specific code paths identified by coverage:

@test()
def test_lookup_funcs_wildcard():
    """lookup_funcs with '*' returns all functions (covers lines 132-134)"""
    result = lookup_funcs("*")
    assert len(result) > 0
    assert result[0]["query"] == "*"

4. Sampling Tests

Test on multiple items for broader coverage:

@test()
def test_callees_multiple():
    """callees works on multiple functions (sampling test)"""
    addrs = get_n_functions()
    result = callees(addrs)
    assert len(result) == len(addrs)

Binary-Specific Tests (Use Sparingly)

Binary-specific tests should only be used when:

1. A code path requires specific binary properties (e.g., C++ mangled names)
2. The edge case cannot be triggered with generic inputs

@test(binary="crackme03.elf")
def test_crackme_specific_edge_case():
    """Tests a code path that requires specific binary structure"""
    # Only runs when crackme03.elf is loaded
    ...

Do NOT use binary-specific tests to verify IDA correctness (e.g., "main is at 0x123e").

Coverage Targets

Module	Target	Notes
api_core	90%+	Core functionality
api_analysis	85%+	Complex decompilation paths
api_memory	85%+	Read/write operations
api_types	80%+	Type system edge cases
api_modify	70%+	Modification operations
api_resources	85%+	Resource URIs
api_stack	80%+	Stack frame operations
api_debug	Skip	Requires active debugger
api_python	Skip	Requires special setup