Search site

Type a word or phrase to search public pages.

    Graphs beyond RAM

    Query large graphs with streaming operators and disk-backed intermediate results.

    Size the host for the work

    A graph should not need to fit into RAM before you can query it. Keep working memory bounded where operators support streaming or disk spill, and use storage for larger intermediate results.

    That makes personal hardware useful for serious data work. It does not remove the need to plan for disk space, computation, or network traffic.

    Workspace-aware routing explains where execution runs.

    Developer reference

    Technical details

    Stream rows; spill blocking operators

    SPARQL execution can emit rows incrementally instead of materializing a complete result set. Operators such as ORDER BY, DISTINCT, GROUP BY, and joins use external sorting or spill paths when their working sets exceed configured budgets. Dictionary access is lazy rather than cloning every term into a process-wide lookup table.

    A sort-merge join can stage inputs on disk and merge ordered streams. Grouping and duplicate elimination also need intermediate state; moving that state to disk changes the memory requirement, not the query's semantics.

    Resource budgets are concrete

    Out-of-core execution is not a guarantee that every operator, correlated subquery, or query plan has constant memory use. Operator budgets also do not certify a hard ceiling for the entire process. Account for buffers, spill storage, and concurrent queries when sizing a Host.

    Blocking operators may need to consume their inputs before producing an answer. A streamed prefix is not evidence that the complete query succeeded: terminal errors remain errors. Cancellation and deadlines provides execution controls for work that should stop.