Cleaning¶

Source: src/wrappers/analytix/cleaning.py

CleaningWrapper is the public cleaning interface exposed through a ContextManager. It provides pandas-like methods for filling missing values, cleaning numeric/categorical/datetime columns, dropping missing or duplicate data, and generating basic data-quality reports.

Users normally call cleaning methods directly on a dataset context returned by an upload operation:

dataset = mf.upload_df(frame)
result = dataset.fillna(column="salary", method="mean")

The same methods are also available from dataset.clean.

The lower-level files are implementation details:

src/core/analytix/cleaning.py builds and executes backend-specific SQL.
src/core/orchestrator/analytix/cleaning.py resolves the active dataset context, detects column type hints, and passes persistence metadata.
src/wrappers/analytix/cleaning.py exposes synchronous and asynchronous public methods.

Public API¶

Every cleaning operation has synchronous and asynchronous forms:

Synchronous	Asynchronous	Purpose
`fillna(column, value=None, method="mean", mapping=None, dtype=None)`	`await afillna(...)`	Fill missing values
`clip(column, lower=None, upper=None)`	`await aclip(...)`	Null values outside numeric bounds
`drop_outliers(column, z_thresh=3.0)`	`await adrop_outliers(...)`	Null z-score outliers
`to_numeric(column)`	`await ato_numeric(...)`	Convert text-like values to numeric
`map_values(column, mapping)`	`await amap_values(...)`	Map categorical values
`filter_valid(column, valid_values)`	`await afilter_valid(...)`	Null values outside an allowed set
`compress_rare(column, min_count=10, other_label="other")`	`await acompress_rare(...)`	Group rare categories
`fix_dates(column)`	`await afix_dates(...)`	Null known invalid date strings
`clip_dates(column, min_dt=None, max_dt=None)`	`await aclip_dates(...)`	Null dates outside a range
`groupby_fillna(column, group_cols, value=None, method="mean", dtype=None)`	`await agroupby_fillna(...)`	Fill missing values by group
`dropna(axis=0, how="any", thresh=None)`	`await adropna(...)`	Drop rows or columns with missing values
`drop(columns=None, axis=0, index=None)`	`await adrop(...)`	Drop rows or columns
`isna()`	`await aisna()`	Boolean null mask
`notna()`	`await anotna()`	Boolean non-null mask
`drop_duplicates(subset=None, keep="first")`	`await adrop_duplicates(...)`	Remove duplicate rows

Public methods return the operation value directly: usually a DataFrame, report dictionary, mask, or None. Invalid operations raise OperationError.

Usage Overview¶

dataset = mf.upload_csv("data/employees.csv")

cleaned_sample = dataset.fillna(column="salary", method="mean")

dataset = await mf.aupload_csv("data/employees.csv")

cleaned_sample = await dataset.afilter_valid(
    column="department",
    valid_values=["Sales", "Engineering", "Finance"],
)

Most cleaning methods materialize a new operation table internally. The public return value is the sample DataFrame; table metadata remains available to the cache and AI execution layers.

Missing Values¶

`fillna`¶

fillna fills missing values in one column. The orchestrator samples the column, detects whether it is numeric, categorical, or datetime, and routes the operation to the matching core implementation. Pass dtype to override detection.

result = dataset.fillna(
    column="salary",
    method="mean",
)

result = await dataset.afillna(
    column="department",
    method="constant",
    value="Unknown",
    dtype="categorical",
)

Parameters:

Parameter	Type	Description
`column`	`str`	Column to clean.
`value`	any	Replacement used by `method="constant"`.
`method`	`str`	Fill strategy. Defaults to `"mean"`.
`mapping`	`dict` or `None`	Mapping used by `method="map"` for categorical columns.
`dtype`	`str` or `None`	Optional override: `"numeric"`, `"categorical"`, or `"datetime"`.

Supported methods depend on detected or supplied dtype:

Dtype	Methods
`numeric`	`mean`, `avg`, `average`, `median`, `mode`, `constant`, `std`, `var`, `min`, `max`, `ffill`, `bfill`
`categorical`	`constant`, `mode`, `map`, `ffill`, `bfill`
`datetime`	`constant`, `min`, `max`, `mean`, `median`, `mode`, `now`, `ffill`, `bfill`

Numeric mean/median/std/var/min/max methods are rejected for categorical columns.

`groupby_fillna`¶

groupby_fillna fills missing values using statistics or fill behavior within groups.

result = dataset.groupby_fillna(
    column="salary",
    group_cols=["department"],
    method="mean",
)

result = await dataset.agroupby_fillna(
    column="category",
    group_cols=["department"],
    method="mode",
    dtype="categorical",
)

Parameters:

Parameter	Type	Description
`column`	`str`	Column to fill.
`group_cols`	`list[str]`	Columns used for grouping. Required.
`value`	any	Replacement value for `method="constant"` where supported.
`method`	`str`	Group-wise fill strategy. Defaults to `"mean"`.
`dtype`	`str` or `None`	Optional dtype override: `"numeric"`, `"categorical"`, or `"datetime"`.

Supported group methods:

Dtype	Methods
`numeric`	`mean`, `avg`, `average`, `median`, `mode`, `constant`, `std`, `var`, `min`, `max`, `ffill`, `bfill`
`categorical`	`mode`, `ffill`, `bfill`
`datetime`	`min`, `max`, `mean`, `median`, `mode`, `ffill`, `bfill`

group_cols must be non-empty.

`dropna`¶

dropna drops rows or columns based on missing values.

rows = dataset.dropna(axis=0, how="any")
columns = dataset.dropna(axis=1, thresh=0.1)

rows = await dataset.adropna(axis=0, how="all")

Parameters:

Parameter	Type	Description
`axis`	`0`, `1`, `"index"`, or `"columns"`	`0`/`"index"` drops rows; `1`/`"columns"` drops columns.
`how`	`"any"` or `"all"`	Used when `thresh` is not provided. `"any"` drops if any value is missing; `"all"` drops only if all values are missing.
`thresh`	`int`, `float`, or `None`	Minimum non-null count. A float between 0 and 1 is treated as a maximum allowed null fraction.

When thresh is provided, how is ignored.

`isna` and `notna`¶

isna and notna return boolean masks as generated tables.

null_mask = dataset.isna()
valid_mask = dataset.notna()

null_mask = await dataset.aisna()
valid_mask = await dataset.anotna()

Numeric Cleaning¶

`clip`¶

clip creates a cleaned numeric column where values outside optional lower and upper bounds become NULL. This is range enforcement, not pandas-style clamping to the boundary value.

result = dataset.clip(column="salary", lower=0, upper=250000)

result = await dataset.aclip(column="score", lower=0, upper=100)

Parameters:

Parameter	Type	Description
`column`	`str`	Numeric column to clean.
`lower`	`int`, `float`, or `None`	Values below this are set to `NULL`.
`upper`	`int`, `float`, or `None`	Values above this are set to `NULL`.

`drop_outliers`¶

drop_outliers creates a cleaned column where values with absolute z-score greater than z_thresh become NULL.

result = dataset.drop_outliers(column="salary", z_thresh=3.0)

result = await dataset.adrop_outliers(column="amount", z_thresh=2.5)

`to_numeric`¶

to_numeric converts a text-like column to a numeric type in place, stripping non-numeric characters from its values. The target column type (integer, float, or decimal) is auto-detected from a sample of the cleaned values. Invalid or empty numeric tokens become NULL.

result = dataset.to_numeric(column="price_text")

result = await dataset.ato_numeric(column="amount_raw")

Categorical Cleaning¶

`map_values`¶

map_values creates a cleaned column by replacing values using a mapping dictionary. Unmapped values keep their original value.

result = dataset.map_values(
    column="status",
    mapping={"Y": "yes", "N": "no"},
)

result = await dataset.amap_values(
    column="segment",
    mapping={"SMB": "small_business"},
)

`filter_valid`¶

filter_valid keeps values that appear in valid_values and sets all other non-null values to NULL.

result = dataset.filter_valid(
    column="department",
    valid_values=["Sales", "Engineering", "Finance"],
)

result = await dataset.afilter_valid(
    column="state",
    valid_values=["CA", "NY", "TX"],
)

valid_values must be non-empty.

`compress_rare`¶

compress_rare replaces categories whose frequency is less than min_count with other_label.

result = dataset.compress_rare(
    column="city",
    min_count=10,
    other_label="other",
)

result = await dataset.acompress_rare(
    column="category",
    min_count=5,
    other_label="rare",
)

Datetime Cleaning¶

`fix_dates`¶

fix_dates handles known invalid date strings such as 0000-00-00 by setting the cleaned value to NULL.

result = dataset.fix_dates(column="signup_date")

result = await dataset.afix_dates(column="event_date")

`clip_dates`¶

clip_dates creates a cleaned date column where dates outside the supplied range become NULL.

result = dataset.clip_dates(
    column="signup_date",
    min_dt="2020-01-01",
    max_dt="2025-12-31",
)

result = await dataset.aclip_dates(column="event_date", min_dt="2023-01-01")

Parameters:

Parameter	Type	Description
`column`	`str`	Date or timestamp column to clean.
`min_dt`	`str` or `None`	Minimum allowed date, inclusive.
`max_dt`	`str` or `None`	Maximum allowed date, inclusive.

If both bounds are omitted, the core operation defaults to 1900-01-01 through 2100-01-01.

Row and Column Removal¶

`drop`¶

drop removes rows or columns and materializes the result as a generated table.

rows_removed = dataset.drop(axis=0, index=[0, 3, 5])
cols_removed = dataset.drop(axis=1, columns=["notes", "raw_value"])

result = await dataset.adrop(axis=1, columns=["temporary_column"])

Parameters:

Parameter	Type	Description
`columns`	`list[str]` or `None`	Columns to drop when `axis=1`.
`axis`	`0`, `1`, `"index"`, or `"columns"`	Drop rows with `0`/`"index"`; drop columns with `1`/`"columns"`.
`index`	`list[int]` or `None`	Zero-based row positions to drop when `axis=0`.

`drop_duplicates`¶

drop_duplicates removes duplicate rows using SQL window functions.

result = dataset.drop_duplicates(subset=["email"], keep="first")

result = await dataset.adrop_duplicates(
    subset=["customer_id", "event_time"],
    keep=False,
)

Parameters:

Parameter	Type	Description
`subset`	`list[str]` or `None`	Columns used to identify duplicates. Defaults to all columns.
`keep`	`"first"`, `"last"`, or `False`	Which duplicate row to keep. `False` keeps only rows with no duplicates.

Return Values and Errors¶

Public cleaning methods return the underlying DataFrame, dictionary, mask, or other simple result. Failed operations raise OperationError. The internal response envelope still carries new_table, column metadata, and fill metrics.

Generated Tables¶

Most cleaning operations are non-destructive to the source upload table. They create a generated table internally, usually in the same active schema.

Column-cleaning operations usually:

Copy the source table to a generated operation table.
Add a generated column such as cleaned_salary_mean_filled.
Populate the generated column while preserving the original column.
Return a sample DataFrame containing the source and generated columns.

Row/column operations such as dropna, drop, isna, notna, and drop_duplicates materialize a query result as the generated table.

Backend Behavior¶

Cleaning supports DuckDB and PostgreSQL adapters:

Identifiers are sanitized and quoted before SQL is generated.
PostgreSQL uses ctid for row-wise update joins where needed.
DuckDB uses rowid for row-wise update joins where needed.
Some forward/backward fill implementations differ by backend because DuckDB supports IGNORE NULLS window expressions and PostgreSQL uses fallback window expressions.
Date and percentile expressions use backend-specific SQL where necessary.

Errors¶

Cleaning methods raise OperationError for invalid input or backend failures.

fillna(method="constant") requires value.
fillna(method="map") requires mapping.
Numeric fill methods such as mean, median, std, var, min, and max are rejected for categorical columns.
groupby_fillna requires group_cols.
dropna rejects invalid axis, invalid how, and non-positive thresh.
drop requires index for row drops and columns for column drops.
drop_duplicates requires keep to be "first", "last", or False.
filter_valid requires a non-empty valid_values list.

API Reference¶

`memframe.wrappers.analytix.cleaning.CleaningWrapper` ¶

Bases: CleaningOrchestrator

Wrapper around CleaningOrchestrator with async/sync method pairs.

Each operation is exposed as: - an async method prefixed with a (for example, afillna) - a sync-friendly counterpart (for example, fillna) decorated with @async_to_sync

`afillna(column, value=None, method='mean', mapping=None, dtype=None)` `async` ¶

Asynchronously fill missing values in a column.

`fillna(column, value=None, method='mean', mapping=None, dtype=None)` `async` ¶

Synchronously fill missing values in a column.

`aclip(column, lower=None, upper=None)` `async` ¶

Asynchronously clip column values to lower/upper bounds.

For date/datetime columns, pass date strings ('YYYY-MM-DD') as bounds.

`clip(column, lower=None, upper=None)` `async` ¶

Synchronously clip column values to lower/upper bounds.

For date/datetime columns, pass date strings ('YYYY-MM-DD') as bounds.

`adrop_outliers(column, z_thresh=3.0)` `async` ¶

Asynchronously drop outlier rows using a z-score threshold.

`drop_outliers(column, z_thresh=3.0)` `async` ¶

Synchronously drop outlier rows using a z-score threshold.

`ato_numeric(column)` `async` ¶

Asynchronously coerce a column to numeric dtype.

`to_numeric(column)` `async` ¶

Synchronously coerce a column to numeric dtype.

`amap_values(column, mapping)` `async` ¶

Asynchronously map column values using a mapping dictionary.

`map_values(column, mapping)` `async` ¶

Synchronously map column values using a mapping dictionary.

`afilter_valid(column, valid_values)` `async` ¶

Asynchronously keep rows whose values are in a valid set.

`filter_valid(column, valid_values)` `async` ¶

Synchronously keep rows whose values are in a valid set.

`acompress_rare(column, min_count=10, other_label='other')` `async` ¶

Asynchronously compress low-frequency categories into one label.

`compress_rare(column, min_count=10, other_label='other')` `async` ¶

Synchronously compress low-frequency categories into one label.

`afix_dates(column)` `async` ¶

Asynchronously parse and normalize date values in a column.

`fix_dates(column)` `async` ¶

Synchronously parse and normalize date values in a column.

`aclip_dates(column, min_dt=None, max_dt=None)` `async` ¶

Asynchronously clip date values to optional min/max bounds.

`clip_dates(column, min_dt=None, max_dt=None)` `async` ¶

Synchronously clip date values to optional min/max bounds.

`agroupby_fillna(column, group_cols, value=None, method='mean', dtype=None)` `async` ¶

Asynchronously fill missing values using group-wise statistics.

`groupby_fillna(column, group_cols, value=None, method='mean', dtype=None)` `async` ¶

Synchronously fill missing values using group-wise statistics.

`adropna(axis=0, how='any', thresh=None)` `async` ¶

Asynchronously drop missing data by axis/how/thresh settings.

`dropna(axis=0, how='any', thresh=None)` `async` ¶

Synchronously drop missing data by axis/how/thresh settings.

`adrop(columns=None, axis=0, index=None)` `async` ¶

Asynchronously drop rows or columns by labels/axis.

`drop(columns=None, axis=0, index=None)` `async` ¶

Synchronously drop rows or columns by labels/axis.

`aisna()` `async` ¶

Asynchronously return NA/null indicator mask.

`isna()` `async` ¶

Synchronously return NA/null indicator mask.

`anotna()` `async` ¶

Asynchronously return non-null indicator mask.

`notna()` `async` ¶

Synchronously return non-null indicator mask.

`adrop_duplicates(subset=None, keep='first')` `async` ¶

Asynchronously remove duplicate rows with keep strategy.

`drop_duplicates(subset=None, keep='first')` `async` ¶

Synchronously remove duplicate rows with keep strategy.

Cleaning¶

Public API¶

Usage Overview¶

Missing Values¶

fillna¶

groupby_fillna¶

dropna¶

isna and notna¶

Numeric Cleaning¶

clip¶

drop_outliers¶

to_numeric¶

Categorical Cleaning¶

map_values¶

filter_valid¶

compress_rare¶

Datetime Cleaning¶

fix_dates¶

clip_dates¶

Row and Column Removal¶

drop¶

drop_duplicates¶

Return Values and Errors¶

Generated Tables¶

Backend Behavior¶

Errors¶

API Reference¶

memframe.wrappers.analytix.cleaning.CleaningWrapper ¶

afillna(column, value=None, method='mean', mapping=None, dtype=None) async ¶

fillna(column, value=None, method='mean', mapping=None, dtype=None) async ¶

aclip(column, lower=None, upper=None) async ¶

clip(column, lower=None, upper=None) async ¶

adrop_outliers(column, z_thresh=3.0) async ¶

drop_outliers(column, z_thresh=3.0) async ¶

ato_numeric(column) async ¶

to_numeric(column) async ¶

amap_values(column, mapping) async ¶

map_values(column, mapping) async ¶

afilter_valid(column, valid_values) async ¶

filter_valid(column, valid_values) async ¶

acompress_rare(column, min_count=10, other_label='other') async ¶

compress_rare(column, min_count=10, other_label='other') async ¶

afix_dates(column) async ¶

fix_dates(column) async ¶

aclip_dates(column, min_dt=None, max_dt=None) async ¶

clip_dates(column, min_dt=None, max_dt=None) async ¶

agroupby_fillna(column, group_cols, value=None, method='mean', dtype=None) async ¶

groupby_fillna(column, group_cols, value=None, method='mean', dtype=None) async ¶

adropna(axis=0, how='any', thresh=None) async ¶

dropna(axis=0, how='any', thresh=None) async ¶

adrop(columns=None, axis=0, index=None) async ¶

drop(columns=None, axis=0, index=None) async ¶

aisna() async ¶

isna() async ¶

anotna() async ¶

notna() async ¶

adrop_duplicates(subset=None, keep='first') async ¶

drop_duplicates(subset=None, keep='first') async ¶

`fillna`¶

`groupby_fillna`¶

`dropna`¶

`isna` and `notna`¶

`clip`¶

`drop_outliers`¶

`to_numeric`¶

`map_values`¶

`filter_valid`¶

`compress_rare`¶

`fix_dates`¶

`clip_dates`¶

`drop`¶

`drop_duplicates`¶

`memframe.wrappers.analytix.cleaning.CleaningWrapper` ¶

`afillna(column, value=None, method='mean', mapping=None, dtype=None)` `async` ¶

`fillna(column, value=None, method='mean', mapping=None, dtype=None)` `async` ¶

`aclip(column, lower=None, upper=None)` `async` ¶

`clip(column, lower=None, upper=None)` `async` ¶

`adrop_outliers(column, z_thresh=3.0)` `async` ¶

`drop_outliers(column, z_thresh=3.0)` `async` ¶

`ato_numeric(column)` `async` ¶

`to_numeric(column)` `async` ¶

`amap_values(column, mapping)` `async` ¶

`map_values(column, mapping)` `async` ¶

`afilter_valid(column, valid_values)` `async` ¶

`filter_valid(column, valid_values)` `async` ¶

`acompress_rare(column, min_count=10, other_label='other')` `async` ¶

`compress_rare(column, min_count=10, other_label='other')` `async` ¶

`afix_dates(column)` `async` ¶

`fix_dates(column)` `async` ¶

`aclip_dates(column, min_dt=None, max_dt=None)` `async` ¶

`clip_dates(column, min_dt=None, max_dt=None)` `async` ¶

`agroupby_fillna(column, group_cols, value=None, method='mean', dtype=None)` `async` ¶

`groupby_fillna(column, group_cols, value=None, method='mean', dtype=None)` `async` ¶

`adropna(axis=0, how='any', thresh=None)` `async` ¶

`dropna(axis=0, how='any', thresh=None)` `async` ¶

`adrop(columns=None, axis=0, index=None)` `async` ¶

`drop(columns=None, axis=0, index=None)` `async` ¶

`aisna()` `async` ¶

`isna()` `async` ¶

`anotna()` `async` ¶

`notna()` `async` ¶

`adrop_duplicates(subset=None, keep='first')` `async` ¶

`drop_duplicates(subset=None, keep='first')` `async` ¶