Skip to content

Inspect

Source: src/memframe/wrappers/analytix/inspection.py

TableOpsWrapper is the public inspection and table-utility interface exposed through a ContextManager. It provides pandas-like methods for previewing rows, reading schema metadata, computing summaries, applying lightweight table utilities, and retrieving compact table properties from the active backend table.

Users normally call inspect methods directly on a dataset context returned by an upload operation:

dataset = mf.upload_df(frame)
result = dataset.head(n=5)

The same methods are also available from dataset.inspect.

The lower-level files are implementation details:

  • src/core/analytix/table_ops.py builds and executes backend-specific SQL.
  • src/core/orchestrator/analytix/table_ops.py resolves the active dataset context and delegates work to the core table engine.
  • src/wrappers/analytix/inspect.py exposes synchronous and asynchronous public methods.

Public API

Every inspect operation has synchronous and asynchronous forms:

Synchronous Asynchronous Purpose
head(n=10, columns=None) await ahead(...) First rows
tail(n=10, columns=None) await atail(...) Last rows
sample(n=10, columns=None, random_state=None) await asample(...) Random rows
info() await ainfo() Per-column table information
describe(columns=None) await adescribe(...) Numeric descriptive statistics
null_analysis(columns=None) await anull_analysis(...) Null distribution by column
data_quality_missing_values(columns) await adata_quality_missing_values(...) Missing-value counts
data_quality_completeness_score(columns) await adata_quality_completeness_score(...) Completeness percentages
comprehensive_numeric_summary(columns) await acomprehensive_numeric_summary(...) Numeric summary report
statistical_profile_report(columns) await astatistical_profile_report(...) Combined profile report
full_table(columns=None, chunk_size=None) await afull_table(...) Full table or chunk iterator
astype(columns=None, dtypes=None, dtype_map=None) await aastype(...) Cast selected columns
insert(column, value) await ainsert(...) Add a column from a value list
map(func, na_action=None, columns=None, datetime_action="skip") await amap(...) Apply SQL expression to values
rename(columns) await arename(...) Rename columns
set_index(columns) await aset_index(...) Add a primary-key index
update(on, other_table, other_schema="upload", overwrite=True, errors="ignore") await aupdate(...) Update from another table
resample(time_column, rule, agg="COUNT", value_column=None, label="left", closed="left") await aresample(...) Time-series aggregation
columns() await acolumns() Column labels
dtypes() await adtypes() Column database types
shape() await ashape() Row and column count
values() await avalues() Table values as nested lists
items() await aitems() Column/value iterator result
iterrows() await aiterrows() Row iterator result
itertuples(index=True) await aitertuples(...) Tuple-style row iterator result

Public methods return DataFrames, dictionaries, scalars, or iterators directly. Invalid operations raise OperationError.

Usage Overview

dataset = mf.upload_csv("data/sales.csv")

frame = dataset.head(n=5, columns=["customer_id", "amount"])

dataset = await mf.aupload_csv("data/sales.csv")

frame = await dataset.adescribe(columns=["amount"])

Inspect methods are exposed directly through context forwarding. You can use dataset.head(...) or the explicit dataset.inspect.head(...) form.

Row Preview

head returns the first n rows from the active table.

result = dataset.head(n=10)

result = await dataset.ahead(
    n=5,
    columns=["customer_id", "amount"],
)

Parameters:

Parameter Type Description
n int Maximum number of rows to return. Defaults to 10.
columns list[str] or None Optional columns to include. Invalid names are ignored; if none are valid, all columns are used.

The public method returns the DataFrame directly. Row/column metadata remains internal to the response envelope.

tail

tail returns the last n rows by counting total rows and applying an offset.

result = dataset.tail(n=5)

result = await dataset.atail(n=5, columns=["name", "score"])

Parameters:

Parameter Type Description
n int Maximum number of rows to return. Defaults to 10.
columns list[str] or None Optional columns to include.

sample

sample returns random rows using backend RANDOM() ordering.

result = dataset.sample(n=10, random_state=42)

result = await dataset.asample(
    n=10,
    columns=["name", "score"],
)

Parameters:

Parameter Type Description
n int Number of random rows to return. Defaults to 10.
columns list[str] or None Optional columns to include.
random_state int or None Optional seed. PostgreSQL applies it through setseed; DuckDB currently accepts but does not use it for deterministic sampling.

full_table

full_table returns all rows as a DataFrame unless chunk_size is provided.

result = dataset.full_table(columns=["id", "name"])

result = await dataset.afull_table(chunk_size=1000)
async for chunk in result:
    ...

Parameters:

Parameter Type Description
columns list[str] or None Optional columns to include.
chunk_size int or None Positive row count per chunk. When set, the method returns an async iterator.

Invalid or non-positive chunk_size raises OperationError.

Summary Methods

info

info returns one row per column with database type, null counts, non-null counts, null percentage, and distinct count.

result = dataset.info()

result = await dataset.ainfo()

Parameters: none.

Return behavior:

  • The method returns a DataFrame with one row per column.
  • Generated summary-table and table-level metadata remain internal when persistence context is available.

describe

describe computes numeric statistics: count, mean, std, min, 25%, 50%, 75%, and max.

result = dataset.describe()

result = await dataset.adescribe(columns=["amount", "score"])

Parameters:

Parameter Type Description
columns list[str] or None Numeric columns to summarize. If omitted, numeric columns are discovered from the backend schema.

The result DataFrame has a statistic column plus one column per summarized numeric column. If no numeric columns are available, the method raises OperationError.

null_analysis

null_analysis reports whether selected columns contain null values and their missing percentages.

result = dataset.null_analysis(columns=["score", "note"])

result = await dataset.anull_analysis()

Parameters:

Parameter Type Description
columns list[str], "*", or None Columns to analyze. None, "*", or ["*"] analyzes all columns.

The result DataFrame is indexed by column name when data is available and contains contains_null and percent_missing columns.

Data Quality Reports

data_quality_missing_values

data_quality_missing_values returns per-column total, non_null, missing, and missing_pct values.

result = dataset.data_quality_missing_values(
    columns=["salary", "department"],
)

result = await dataset.adata_quality_missing_values(
    columns=["salary", "department"],
)

data_quality_completeness_score

data_quality_completeness_score returns completeness percentages for each requested column.

result = dataset.data_quality_completeness_score(columns=["salary"])

result = await dataset.adata_quality_completeness_score(columns=["salary"])

comprehensive_numeric_summary

comprehensive_numeric_summary generates numeric summaries for up to the first 20 requested columns.

result = dataset.comprehensive_numeric_summary(columns=["salary", "bonus"])

result = await dataset.acomprehensive_numeric_summary(columns=["salary", "bonus"])

statistical_profile_report

statistical_profile_report combines completeness scoring and numeric summary output into one response.

result = dataset.statistical_profile_report(columns=["salary", "bonus"])

result = await dataset.astatistical_profile_report(columns=["salary", "bonus"])

Table Utilities

astype

astype casts selected columns and returns only the casted columns.

result = dataset.astype(
    dtype_map={"amount": "float64", "customer_id": "str"},
)

result = await dataset.aastype(
    columns=["amount"],
    dtypes=["float64"],
)

Parameters:

Parameter Type Description
columns list[str] or None Columns to cast when dtype_map is not provided.
dtypes list[str] or None Target dtypes matching columns by position.
dtype_map dict[str, str] or None Direct mapping of column name to target dtype. Takes precedence over columns/dtypes.

Supported dtype aliases:

Alias group Examples SQL target
Integer int, int8, int16, int32 INTEGER
Big integer int64 BIGINT
Float float, float32 FLOAT
Double float64, double DOUBLE
Text str, string, text TEXT

insert

insert adds a new text column and fills it from a list of values. The list length must match the row count.

result = dataset.insert(
    column="segment",
    value=["retail", "enterprise", "retail"],
)

result = await dataset.ainsert(
    column="segment",
    value=["retail", "enterprise", "retail"],
)

Parameters:

Parameter Type Description
column str New column name.
value list Values assigned row-by-row. Must match row count.

map

map applies a SQL expression to compatible columns. Use x as the placeholder for the current column expression.

result = dataset.map(
    func="x * 100",
    columns=["conversion_rate"],
)

result = await dataset.amap(
    func="UPPER(x)",
    columns=["segment"],
    na_action="ignore",
)

Parameters:

Parameter Type Description
func str SQL expression using x as the current column placeholder.
na_action str or None "ignore" preserves nulls; None applies the expression normally.
columns list[str], "*", or None Columns to map. Defaults to all columns.
datetime_action str How datetime columns are handled. Defaults to "skip".

Supported datetime_action values:

Value Behavior
skip Skip datetime columns.
cast_string Cast datetime values to text before applying func.
extract_epoch Apply func to epoch seconds.
keep Return datetime column unchanged.
error Return an error if a datetime column is selected.

Numeric columns accept arithmetic expressions. String columns are only applied when the expression uses string-safe functions such as UPPER, LOWER, LENGTH, or TRIM. Boolean columns are auto-cast to integer for arithmetic expressions.

rename

rename renames columns in place and returns a current table preview.

result = dataset.rename(columns={"old_name": "new_name"})

result = await dataset.arename(columns={"old_name": "new_name"})

Parameters:

Parameter Type Description
columns dict[str, str] Mapping of old column names to new names.

set_index

set_index adds a primary-key constraint over selected columns.

result = dataset.set_index(columns=["customer_id"])

set_index parameters:

Parameter Type Description
columns list[str] Columns used for the primary key.

update

update updates rows from another backend table using a key column.

result = dataset.update(
    on="customer_id",
    other_table="customer_updates",
    other_schema="upload",
)

result = await dataset.aupdate(
    on="customer_id",
    other_table="customer_updates",
    overwrite=True,
    errors="ignore",
)

Parameters:

Parameter Type Description
on str Key column used to match rows.
other_table str Table containing update values.
other_schema str Schema for other_table. Defaults to "upload".
overwrite bool Whether matched values should overwrite existing values.
errors str Error handling mode passed to the core update implementation. Defaults to "ignore".

resample

resample groups timestamp data into time buckets and applies an aggregate.

result = dataset.resample(
    time_column="event_time",
    rule="day",
    agg="COUNT",
)

result = await dataset.aresample(
    time_column="event_time",
    rule="month",
    agg="SUM",
    value_column="amount",
)

Parameters:

Parameter Type Description
time_column str Date/timestamp column used for bucketing.
rule str Time bucket rule passed to the core SQL implementation.
agg str Aggregate function. Defaults to "COUNT".
value_column str or None Column aggregated for value-based aggregates.
label str Bucket labeling option. Defaults to "left".
closed str Bucket boundary option. Defaults to "left".

Property Methods

These methods return compact dictionary or list payloads directly:

Method Async Public result
columns() acolumns() [...]
dtypes() adtypes() {column: db_type}
shape() ashape() {"shape": (rows, columns)}
values() avalues() {"values": [[...], ...]} 
print(dataset.shape()["shape"])
print(dataset.columns())

shape = await dataset.ashape()
dtypes = await dataset.adtypes()

Parameters: none for these property methods.

Iterator Methods

items, iterrows, and itertuples mirror pandas iterator-style APIs. The wrapper exposes synchronous and asynchronous forms:

items = dataset.items()
rows = dataset.iterrows()
tuples = dataset.itertuples(index=False)

items = await dataset.aitems()
rows = await dataset.aiterrows()
tuples = await dataset.aitertuples(index=False)

Parameters:

Method Parameter Type Description
items none - Returns column-oriented iterator payload from the core engine.
iterrows none - Returns row-oriented iterator payload from the core engine.
itertuples index bool Whether tuple rows include an index value. Defaults to True.

The public method returns the exact payload or iterator from the core table engine.

Return Values and Errors

Public inspection methods return DataFrames, dictionaries, scalars, or async iterators directly. Generated-table and result metadata remains internal. Failed operations raise OperationError.

Generated Tables

Some inspect methods create generated summary tables internally when method-call logging receives backend context. This is most visible for info, describe, null_analysis, and sample.

Preview methods such as head, tail, sample, and unchunked full_table are read-oriented and return a DataFrame sample directly.

Backend Behavior

Inspect supports DuckDB and PostgreSQL adapters:

  • Both backends use quoted identifiers and schema-aware table names.
  • Schema discovery is delegated to the active database adapter.
  • describe uses backend-specific percentile functions.
  • sample uses backend RANDOM() ordering.
  • PostgreSQL can report relation memory usage through info; DuckDB currently returns None for memory_usage.
  • Column and table identifiers are sanitized before SQL is generated.

Errors

Inspect methods raise OperationError for invalid input or backend failures.

  • Invalid chunk_size in full_table raises OperationError.
  • describe raises an error when no numeric columns are available.
  • astype raises an error for missing columns or unsupported dtype aliases.
  • astype requires either dtype_map or matching columns and dtypes.
  • insert raises an error when value is not a list or its length does not match the row count.
  • map raises an error when func is not a SQL expression string or when no selected columns are compatible with the expression.
  • rename, set_index, update, and resample can raise backend SQL errors when identifiers or constraints are invalid.

API Reference

memframe.wrappers.analytix.inspection.TableOpsWrapper

Bases: TableOpsOrchestrator

Wrapper around TableOpsOrchestrator with async/sync methods.

ahead(n=10, columns=None) async

Asynchronously return the first n rows.

head(n=10, columns=None) async

Synchronously return the first n rows.

atail(n=10, columns=None) async

Asynchronously return the last n rows.

tail(n=10, columns=None) async

Synchronously return the last n rows.

asample(n=10, columns=None, random_state=None) async

Asynchronously sample n rows from the table.

sample(n=10, columns=None, random_state=None) async

Synchronously sample n rows from the table.

ainfo() async

Asynchronously return dataset information summary.

info() async

Synchronously return dataset information summary.

adescribe(columns=None) async

Asynchronously compute descriptive statistics.

describe(columns=None) async

Synchronously compute descriptive statistics.

anull_analysis(columns=None) async

Asynchronously analyze null distribution across columns.

null_analysis(columns=None) async

Synchronously analyze null distribution across columns.

afull_table(columns=None, chunk_size=None) async

Asynchronously return full table data, optionally chunked.

full_table(columns=None, chunk_size=None) async

Synchronously return full table data, optionally chunked.

aastype(columns=None, dtypes=None, dtype_map=None) async

Asynchronously cast columns to target dtypes.

astype(columns=None, dtypes=None, dtype_map=None) async

Synchronously cast columns to target dtypes.

ainsert(column, value) async

Asynchronously insert or assign a column value.

insert(column, value) async

Synchronously insert or assign a column value.

amap(func, na_action=None, columns=None, datetime_action='skip') async

Asynchronously apply a mapping function to values.

map(func, na_action=None, columns=None, datetime_action='skip') async

Synchronously apply a mapping function to values.

arename(columns) async

Asynchronously rename columns using a mapping.

rename(columns) async

Synchronously rename columns using a mapping.

aset_index(columns) async

Asynchronously set one or more columns as index.

set_index(columns) async

Synchronously set one or more columns as index.

aupdate(on, other_table, other_schema='upload', overwrite=True, errors='ignore') async

Asynchronously update rows from another table using a key.

update(on, other_table, other_schema='upload', overwrite=True, errors='ignore') async

Synchronously update rows from another table using a key.

aresample(time_column, rule, agg='COUNT', value_column=None, label='left', closed='left') async

Asynchronously resample time-series data with aggregation.

resample(time_column, rule, agg='COUNT', value_column=None, label='left', closed='left') async

Synchronously resample time-series data with aggregation.

acolumns() async

Asynchronously return column labels.

columns() async

Synchronously return column labels.

adtypes() async

Asynchronously return column dtypes.

dtypes() async

Synchronously return column dtypes.

ashape() async

Asynchronously return table shape.

shape() async

Synchronously return table shape.

avalues() async

Asynchronously return table values.

values() async

Synchronously return table values.

aitems() async

Asynchronously iterate over column name/value pairs.

items() async

Synchronously iterate over column name/value pairs.

aiterrows() async

Asynchronously iterate over rows as index/series pairs.

iterrows() async

Synchronously iterate over rows as index/series pairs.

aitertuples(index=True) async

Asynchronously iterate rows as named tuples.

itertuples(index=True) async

Synchronously iterate rows as named tuples.