Clean¶
Source: src/memframe/wrappers/analytix/cleaning.py
CleaningWrapper (Clean — formerly Cleaning) is the public cleaning interface exposed through a
ContextManager. It provides pandas-like methods for filling missing values,
cleaning numeric/categorical/datetime columns, dropping missing or duplicate
data, and generating basic data-quality reports.
Users normally call cleaning methods directly on a dataset context returned by an upload operation:
The same methods are also available from dataset.clean.
The lower-level files are implementation details:
src/core/analytix/cleaning.pybuilds and executes backend-specific SQL.src/core/orchestrator/analytix/cleaning.pyresolves the active dataset context, detects column type hints, and passes persistence metadata.src/wrappers/analytix/cleaning.pyexposes synchronous and asynchronous public methods.
Public API¶
Every cleaning operation has synchronous and asynchronous forms:
| Synchronous | Asynchronous | Purpose |
|---|---|---|
fillna(column, value=None, method="mean", mapping=None, dtype=None) |
await afillna(...) |
Fill missing values |
interpolate(column, method="linear", limit=None, limit_direction="forward", limit_area=None, order_by=None, dtype=None) |
await ainterpolate(...) |
Fill missing values by linear interpolation |
replace(column=None, to_replace=None, value=None, regex=False) |
await areplace(...) |
Replace values in one, several, or all columns |
clip(column, lower=None, upper=None) |
await aclip(...) |
Null values outside numeric bounds |
drop_outliers(column, z_thresh=3.0) |
await adrop_outliers(...) |
Null z-score outliers |
to_numeric(column) |
await ato_numeric(...) |
Convert text-like values to numeric |
map_values(column, mapping) |
await amap_values(...) |
Map categorical values |
filter_valid(column, valid_values) |
await afilter_valid(...) |
Null values outside an allowed set |
compress_rare(column, min_count=10, other_label="other") |
await acompress_rare(...) |
Group rare categories |
fix_dates(column) |
await afix_dates(...) |
Null known invalid date strings |
clip_dates(column, min_dt=None, max_dt=None) |
await aclip_dates(...) |
Null dates outside a range |
groupby_fillna(column, group_cols, value=None, method="mean", dtype=None) |
await agroupby_fillna(...) |
Fill missing values by group |
dropna(axis=0, how="any", thresh=None) |
await adropna(...) |
Drop rows or columns with missing values |
drop(columns=None, axis=0, index=None) |
await adrop(...) |
Drop rows or columns |
isna() |
await aisna() |
Boolean null mask |
notna() |
await anotna() |
Boolean non-null mask |
drop_duplicates(subset=None, keep="first") |
await adrop_duplicates(...) |
Remove duplicate rows |
duplicated(subset=None, keep="first") |
await aduplicated(...) |
Flag duplicate rows with a boolean column |
Public methods return the operation value directly: usually a DataFrame, report
dictionary, mask, or None. Invalid operations raise OperationError.
Usage Overview¶
dataset = mf.upload_csv("data/employees.csv")
cleaned_sample = dataset.fillna(column="salary", method="mean")
dataset = await mf.aupload_csv("data/employees.csv")
cleaned_sample = await dataset.afilter_valid(
column="department",
valid_values=["Sales", "Engineering", "Finance"],
)
Most cleaning methods materialize a new operation table internally. The public return value is the sample DataFrame; table metadata remains available to the cache and AI execution layers.
Missing Values¶
fillna¶
fillna fills missing values in one column. The orchestrator samples the
column, detects whether it is numeric, categorical, or datetime, and routes the
operation to the matching core implementation. Pass dtype to override
detection.
result = await dataset.afillna(
column="department",
method="constant",
value="Unknown",
dtype="categorical",
)
Parameters:
| Parameter | Type | Description |
|---|---|---|
column |
str |
Column to clean. |
value |
any | Replacement used by method="constant". |
method |
str |
Fill strategy. Defaults to "mean". |
mapping |
dict or None |
Mapping used by method="map" for categorical columns. |
dtype |
str or None |
Optional override: "numeric", "categorical", or "datetime". |
Supported methods depend on detected or supplied dtype:
| Dtype | Methods |
|---|---|
numeric |
mean, avg, average, median, mode, constant, std, var, min, max, ffill, bfill |
categorical |
constant, mode, map, ffill, bfill |
datetime |
constant, min, max, mean, median, mode, now, ffill, bfill |
Numeric mean/median/std/var/min/max methods are rejected for
categorical columns.
groupby_fillna¶
groupby_fillna fills missing values using statistics or fill behavior within
groups.
result = await dataset.agroupby_fillna(
column="category",
group_cols=["department"],
method="mode",
dtype="categorical",
)
Parameters:
| Parameter | Type | Description |
|---|---|---|
column |
str |
Column to fill. |
group_cols |
list[str] |
Columns used for grouping. Required. |
value |
any | Replacement value for method="constant" where supported. |
method |
str |
Group-wise fill strategy. Defaults to "mean". |
dtype |
str or None |
Optional dtype override: "numeric", "categorical", or "datetime". |
Supported group methods:
| Dtype | Methods |
|---|---|
numeric |
mean, avg, average, median, mode, constant, std, var, min, max, ffill, bfill |
categorical |
mode, ffill, bfill |
datetime |
min, max, mean, median, mode, ffill, bfill |
group_cols must be non-empty.
interpolate¶
interpolate fills missing values in a numeric column by linear
interpolation along row order (a + (b−a)·(p−p₀)/(p₁−p₀) between the
surrounding valid values). Only method="linear" is supported.
Interior gaps interpolate; edge NaNs fill flat from
the nearest valid value according to limit_direction (forward fills
trailing, backward fills leading, both fills both; leading NaNs stay NaN
under the default forward).
result = await dataset.ainterpolate(
column="temp",
limit=1,
limit_direction="both",
limit_area="inside",
)
Parameters:
| Parameter | Type | Description |
|---|---|---|
column |
str |
Numeric column to fill. |
method |
str |
Must be "linear". Anything else fails loudly. |
limit |
int or None |
Max consecutive NaNs to fill, counted from the fill edge per direction. Must be positive. |
limit_direction |
str |
"forward" (default), "backward", or "both". |
limit_area |
str or None |
None (default, no restriction), "inside" (gaps only), "outside" (edges only). |
order_by |
str, list[str], or None |
Column(s) sequencing rows before interpolating (single or list, ASC, NULLS LAST). Omitted means physical row order. The ordering is temporary: values compute along the sequence but rows keep their original positions. |
dtype |
str or None |
Optional override: only "numeric" is accepted; anything else fails. |
The result column is cleaned_<column>_interpolated (always floating point —
pandas upcasts the same way). Non-numeric columns fail; pass
dtype="numeric" only to override a misdetected numeric column.
order_by accepts one column or a list. Duplicate order keys make ties
nondeterministic — pass a unique key when it matters. Unknown columns fail
before any table is created.
replace¶
replace substitutes values in one column, a list of columns, or all columns
(column=None, the pandas default). to_replace takes pandas' scalar, list,
flat-dict, nested-dict, and pattern forms; value is the replacement (None
writes NULL); regex=True matches patterns on text columns instead of exact
equality.
result = await dataset.areplace(to_replace=r"^ba.$", value="new", regex=True)
result = await dataset.areplace(to_replace={"A": {0: 100, 4: 400}})
Parameters:
| Parameter | Type | Description |
|---|---|---|
column |
str, list[str], or None |
Target column(s). None means all columns (generated cleaned_ columns excluded). |
to_replace |
scalar, list, dict, pattern, or None |
Values/patterns to find. Dicts follow pandas' rule: with value given, keys are column names; without it, {old: new} (or nested {col: {old: new}}). |
value |
scalar, list, or None |
Replacement. Lists must match to_replace length; None writes NULL. |
regex |
bool, pattern, or list/dict of patterns |
True interprets string entries as patterns; a pattern (or list/dict) implies it with to_replace=None. |
Each replaced column gains a cleaned_<column>_replaced column; numerics
always match exactly, even under regex=True. Replacements are cast to the
column's native type (a value that cannot live there fails loudly — scope
column= to dodge frame-wide mismatches), except regex replacements, which
stay raw text so backreferences pass through. Frame-wide regex skips non-text
columns, mirroring pandas; naming a non-text column explicitly fails.
inplace is not offered — operations return new columns, never rewrites.
dropna¶
dropna drops rows or columns based on missing values.
Parameters:
| Parameter | Type | Description |
|---|---|---|
axis |
0, 1, "index", or "columns" |
0/"index" drops rows; 1/"columns" drops columns. |
how |
"any" or "all" |
Used when thresh is not provided. "any" drops if any value is missing; "all" drops only if all values are missing. |
thresh |
int, float, or None |
Minimum non-null count. A float between 0 and 1 is treated as a maximum allowed null fraction. |
When thresh is provided, how is ignored.
isna and notna¶
isna and notna return boolean masks as generated tables.
Numeric Cleaning¶
clip¶
clip creates a cleaned numeric column where values outside optional lower and
upper bounds become NULL. This is range enforcement, not pandas-style
clamping to the boundary value.
Parameters:
| Parameter | Type | Description |
|---|---|---|
column |
str |
Numeric column to clean. |
lower |
int, float, or None |
Values below this are set to NULL. |
upper |
int, float, or None |
Values above this are set to NULL. |
drop_outliers¶
drop_outliers creates a cleaned column where values with absolute z-score
greater than z_thresh become NULL.
to_numeric¶
to_numeric converts a text-like column to a numeric type in place,
stripping non-numeric characters from its values. The target column type
(integer, float, or decimal) is auto-detected from a sample of the cleaned
values. Invalid or empty numeric tokens become NULL.
Categorical Cleaning¶
map_values¶
map_values creates a cleaned column by replacing values using a mapping
dictionary. Unmapped values keep their original value.
filter_valid¶
filter_valid keeps values that appear in valid_values and sets all other
non-null values to NULL.
result = dataset.filter_valid(
column="department",
valid_values=["Sales", "Engineering", "Finance"],
)
valid_values must be non-empty.
compress_rare¶
compress_rare replaces categories whose frequency is less than min_count
with other_label.
Datetime Cleaning¶
fix_dates¶
fix_dates handles known invalid date strings such as 0000-00-00 by setting
the cleaned value to NULL.
clip_dates¶
clip_dates creates a cleaned date column where dates outside the supplied
range become NULL.
Parameters:
| Parameter | Type | Description |
|---|---|---|
column |
str |
Date or timestamp column to clean. |
min_dt |
str or None |
Minimum allowed date, inclusive. |
max_dt |
str or None |
Maximum allowed date, inclusive. |
If both bounds are omitted, the core operation defaults to 1900-01-01 through
2100-01-01.
Row and Column Removal¶
drop¶
drop removes rows or columns and materializes the result as a generated table.
rows_removed = dataset.drop(axis=0, index=[0, 3, 5])
cols_removed = dataset.drop(axis=1, columns=["notes", "raw_value"])
Parameters:
| Parameter | Type | Description |
|---|---|---|
columns |
list[str] or None |
Columns to drop when axis=1. |
axis |
0, 1, "index", or "columns" |
Drop rows with 0/"index"; drop columns with 1/"columns". |
index |
list[int] or None |
Zero-based row positions to drop when axis=0. |
drop_duplicates¶
drop_duplicates removes duplicate rows using SQL window functions.
Parameters:
| Parameter | Type | Description |
|---|---|---|
subset |
list[str] or None |
Columns used to identify duplicates. Defaults to all columns. |
keep |
"first", "last", or False |
Which duplicate row to keep. False keeps only rows with no duplicates. |
duplicated¶
duplicated flags duplicate rows without removing them: same subset/keep
semantics as drop_duplicates, but the generated table keeps every row plus
a duplicated boolean column. NULLs group together (pandas' NaN == NaN
rule) on all backends, and first/last follow physical row order.
Parameters:
| Parameter | Type | Description |
|---|---|---|
subset |
str, list[str], or None |
Columns used to identify duplicates. Defaults to all columns. |
keep |
"first", "last", or False |
Which occurrence is not flagged. False flags every row of a duplicate set. |
Return Values and Errors¶
Public cleaning methods return the underlying DataFrame, dictionary, mask, or
other simple result. Failed operations raise OperationError. The internal
response envelope still carries new_table, column metadata, and fill metrics.
Generated Tables¶
Most cleaning operations are non-destructive to the source upload table. They create a generated table internally, usually in the same active schema (or database on ClickHouse).
Column-cleaning operations usually:
- Copy the source table to a generated operation table.
- Add a generated column such as
cleaned_salary_mean_filled. - Populate the generated column while preserving the original column.
- Return a sample DataFrame containing the source and generated columns.
Row/column operations such as dropna, drop, isna, notna, and
drop_duplicates materialize a query result as the generated table.
Backend Behavior¶
Cleaning supports DuckDB and PostgreSQL adapters:
- Identifiers are sanitized and quoted before SQL is generated.
- PostgreSQL uses
ctidfor row-wise update joins where needed. - DuckDB uses
rowidfor row-wise update joins where needed. - Some forward/backward fill implementations differ by backend because DuckDB
supports
IGNORE NULLSwindow expressions and PostgreSQL uses fallback window expressions.interpolateinstead uses the portable islands idiom (COUNT+FIRST_VALUEwindows) on all backends; only the float cast differs (DOUBLE/DOUBLE PRECISION/Float64). ClickHouse rebuilds via temp tables because window functions are illegal inALTER TABLE ... UPDATE. - Date and percentile expressions use backend-specific SQL where necessary.
Errors¶
Cleaning methods raise OperationError for invalid input or backend failures.
fillna(method="constant")requiresvalue.fillna(method="map")requiresmapping.- Numeric fill methods such as
mean,median,std,var,min, andmaxare rejected for categorical columns. groupby_fillnarequiresgroup_cols.dropnarejects invalidaxis, invalidhow, and non-positivethresh.droprequiresindexfor row drops andcolumnsfor column drops.drop_duplicatesrequireskeepto be"first","last", orFalse.duplicatedrequireskeepto be"first","last", orFalse, andsubsetto be a label or list of known columns.filter_validrequires a non-emptyvalid_valueslist.interpolaterequiresmethod="linear", a numeric column, a positivelimit,limit_directionin"forward"/"backward"/"both", andlimit_areainNone/"inside"/"outside".replacerejects length-mismatched lists,valuedicts (use nestedto_replace), mixed nested/flat dicts, regex on explicitly named non-text columns, and values that cannot cast to the column type.
API Reference¶
memframe.wrappers.analytix.cleaning.CleaningWrapper
¶
Bases: CleaningOrchestrator
Wrapper around CleaningOrchestrator with async/sync method pairs.
Each operation is exposed as:
- an async method prefixed with a (for example, afillna)
- a sync-friendly counterpart (for example, fillna) decorated with
@async_to_sync
afillna(column, value=None, method='mean', mapping=None, dtype=None)
async
¶
Asynchronously fill missing values in a column.
fillna(column, value=None, method='mean', mapping=None, dtype=None)
async
¶
Synchronously fill missing values in a column.
ainterpolate(column, method='linear', limit=None, limit_direction='forward', limit_area=None, order_by=None, dtype=None)
async
¶
Asynchronously fill nulls by linear interpolation.
interpolate(column, method='linear', limit=None, limit_direction='forward', limit_area=None, order_by=None, dtype=None)
async
¶
Synchronously fill nulls by linear interpolation.
areplace(column=None, to_replace=None, value=None, regex=False)
async
¶
Asynchronously replace values in one, several, or all columns.
replace(column=None, to_replace=None, value=None, regex=False)
async
¶
Synchronously replace values in one, several, or all columns.
aclip(column, lower=None, upper=None)
async
¶
Asynchronously clip column values to lower/upper bounds.
For date/datetime columns, pass date strings ('YYYY-MM-DD') as bounds.
clip(column, lower=None, upper=None)
async
¶
Synchronously clip column values to lower/upper bounds.
For date/datetime columns, pass date strings ('YYYY-MM-DD') as bounds.
adrop_outliers(column, z_thresh=3.0)
async
¶
Asynchronously drop outlier rows using a z-score threshold.
drop_outliers(column, z_thresh=3.0)
async
¶
Synchronously drop outlier rows using a z-score threshold.
ato_numeric(column)
async
¶
Asynchronously coerce a column to numeric dtype.
to_numeric(column)
async
¶
Synchronously coerce a column to numeric dtype.
amap_values(column, mapping)
async
¶
Asynchronously map column values using a mapping dictionary.
map_values(column, mapping)
async
¶
Synchronously map column values using a mapping dictionary.
afilter_valid(column, valid_values)
async
¶
Asynchronously keep rows whose values are in a valid set.
filter_valid(column, valid_values)
async
¶
Synchronously keep rows whose values are in a valid set.
acompress_rare(column, min_count=10, other_label='other')
async
¶
Asynchronously compress low-frequency categories into one label.
compress_rare(column, min_count=10, other_label='other')
async
¶
Synchronously compress low-frequency categories into one label.
afix_dates(column)
async
¶
Asynchronously parse and normalize date values in a column.
fix_dates(column)
async
¶
Synchronously parse and normalize date values in a column.
aclip_dates(column, min_dt=None, max_dt=None)
async
¶
Asynchronously clip date values to optional min/max bounds.
clip_dates(column, min_dt=None, max_dt=None)
async
¶
Synchronously clip date values to optional min/max bounds.
agroupby_fillna(column, group_cols, value=None, method='mean', dtype=None)
async
¶
Asynchronously fill missing values using group-wise statistics.
groupby_fillna(column, group_cols, value=None, method='mean', dtype=None)
async
¶
Synchronously fill missing values using group-wise statistics.
adropna(axis=0, how='any', thresh=None)
async
¶
Asynchronously drop missing data by axis/how/thresh settings.
dropna(axis=0, how='any', thresh=None)
async
¶
Synchronously drop missing data by axis/how/thresh settings.
adrop(columns=None, axis=0, index=None)
async
¶
Asynchronously drop rows or columns by labels/axis.
drop(columns=None, axis=0, index=None)
async
¶
Synchronously drop rows or columns by labels/axis.
aisna()
async
¶
Asynchronously return NA/null indicator mask.
isna()
async
¶
Synchronously return NA/null indicator mask.
anotna()
async
¶
Asynchronously return non-null indicator mask.
notna()
async
¶
Synchronously return non-null indicator mask.
adrop_duplicates(subset=None, keep='first')
async
¶
Asynchronously remove duplicate rows with keep strategy.
drop_duplicates(subset=None, keep='first')
async
¶
Synchronously remove duplicate rows with keep strategy.
aduplicated(subset=None, keep='first')
async
¶
Asynchronously flag duplicate rows with a boolean mask column.
duplicated(subset=None, keep='first')
async
¶
Synchronously flag duplicate rows with a boolean mask column.