Skip to main content

Python API

Zeno has 4 primary decorator functions: model, metric, distill, and inference.

You can pass any number of files with decorated functions to Zeno, but you must have only one model.

ZenoOptions

Every Zeno function is passed a ZenoOptions object with the following parameters. The column values refer to column names in the DataFrame and the paths refer to directories relative to the Zeno configuration file.

@dataclass
class ZenoOptions:
"""Parameters passed to Zeno test functions.

Args:
id_column (str): Column in dataframe with unique identifiers.
data_column (str): Column in dataframe with either raw data or path to data.
label_column (str): Column in dataframe with
either raw labels or path to labels.
output_column (str): Column in dataframe with a given
model's raw output or path to output
data_path (str): Path to directory with data files.
label_path (str): Path to directory with label files.
output_path (str): Path to directory with a given model's output.
distill_columns (map[str, str]): Map from distiller name to distill column.
"""

id_column: str
data_column: str
label_column: str
output_column: str
distill_columns: Dict[str, str]
data_path: str
label_path: str
output_path: str

Model

model functions should return a function that returns a list of model outputs for a given model name.

The function returned by model should take two parameters: a Pandas DataFrame and a ZenoOptions object.

@model
def model(model_path: Path) -> Callable[[df: DataFrame, ops: ZenoOptions], ModelReturn]

ModelReturn

The function should return the following object:

class ModelReturn(BaseModel):
"""Return type for model functions.

Args:
model_output (Series | List): Model output for each sample.
embedding (Series | List[List[float]] | List[NDArray] | NDArray | None):
High-dimensional embedding for each sample. Optional.
other_returns (Dict[str, Series | List] | None): Other returns from the model
to be shown as metadata columns in the UI.
"""

model_output: Union[Series, List[Any]]
embedding: Union[Series, List[List[float]], List[NDArray], NDArray, None] = None
other_returns: Union[Dict[str, Union[Series, List[Any]]], None] = None

Example

Load mock model and return outputs
@model
def model_fn(model_path):
mod = load_model(model_path)
def pred(df: DataFrame, ops: ZenoOptions):
return ModelReturn(model_output=mod(df[ops.data_column]))
return pred

Metric

Functions with the metric decorator return a continuous number given a subset of data. Metrics can be classic functions such as accuracy, or specific measures such as word prevalence.

@metric
def metric_func(df: pd.DataFrame, ops: ZenoOptions) -> MetricReturn:

MetricReturn

It should return the following:

class MetricReturn(BaseModel):
"""Return type for metric functions.

Args:
metric (float): Average metric over subset of data
variance (float): Variance of metric over subset of data
"""

metric: float
variance: Union[float, None] = None

Example

Calculate accuracy of model
@metric
def accuracy(df, ops):
return MetricReturn(metric=100 * (df[ops.label_column] == df[ops.output_column]).sum() / len(df))

Distill

distill functions return a derived metadata column from input data and/or model outputs.

@distill
def distill_fn(df: pd.DataFrame, ops: ZenoOptions) -> DistillReturn:

DistillReturn

They should return the following object:

class DistillReturn(BaseModel):
"""Return type for distill functions

Args:
distill_output (Series | List): Distill outputs for each sample.
"""

distill_output: Union[Series, List[Any]]

Example

Get amplitude of sound file
@distill
def amplitude(df, ops: ZenoOptions):
files = [os.path.join(ops.data_path, f) for f in df[ops.data_column]]
amps = []
for audio in files:
y, _ = librosa.load(audio)
amps.append(np.abs(y).mean())
return DistillReturn(distill_output=amps)

Inference

You can create a Gradio interface for your model by adding an inference decorator. This interface allows you to upload new instances and interactively test your model.

@inference
def inference_fn(ops: ZenoOptions) -> InferenceReturn

InferenceReturn

It should return the following:

class InferenceReturn(BaseModel):
"""Return type for inference UI functions.
Take a look at the Gradio documentation for details.

Args:
input_components (List[Blocks]): List of Gradio input components.
output_components (Blocks): Gradio output component.
input_columns (List[str]): List of input column names matching the
input components.
"""

input_components: List[IOComponent]
output_component: IOComponent
input_columns: List[str]

Example

Create Gradio interface for image classification model
@inference
def gradio_inference(ops: ZenoOptions):
return InferenceReturn(
input_components=[gr.Image(type="filepath")],
output_component=gr.Text(label="Output"),
input_columns=[ops.data_column],
)