Skip to main content

Fabrics

Layer enables you to run your MLOps pipelines effortlessly. These pipelines might require complex environment setups such as specific software and hardware configurations. We call these environments fabrics.

Fabrics enables you to define the following configurations that you need:

  • Hardware: Resource types such as:
    • CPU and RAM of the machines
    • Single/multiple node clusters
    • We are also planning to add support for GPU configurations

Predefined fabrics

Layer provides predefined fabrics with general purpose resource sets. These fabrics are:

Fabric nameDriver resourceWorker resourceNumber of workersSpark versionPython version
f-smallS23
f-mediumS33
f-spark-xxsmallS1S113.1.23
f-spark-xsmallS2S223.1.23
f-spark-smallS3S323.1.23
f-spark-mediumS3S343.1.23
f-spark-largeS3S383.1.23

Parameters

Fabric parameters are defined below.

Fabric name

Name of the fabric. You can use this name in the YAML files (dataset.yml, model.yml) to use this fabric as your computation environment.

Driver and worker resources

Driver and worker resources define the hardware specifications of the fabrics. Layer fabrics have three types of resources:

Resource nameCPUMemory
S111GB
S224GB
S3314GB
note

Driver and workers are terms from Apache Spark project. However, if non-Spark fabric is used, the same terms can still be applied: it's just that there will only be one main server, i.e. driver, and no workers.

Number of workers

In multi-cluster fabrics, this parameter defines how many worker are needed.

Spark version

The Spark version of the fabric.

Python Version

The Python version of the fabric.

How to use Fabrics

You can pass your choice of fabric, to run their entity computation on. We support fabric configuration in the respective YAML files (model YAML, dataset YAML) in the following format:

fabric: [FABRIC-NAME]

Fabrics for feature sets

You can use any of the fabrics to build your feature sets in two ways:

Globally

Global fabric config for all features as a root key in the YML file like:

apiVersion: 1
type: featureset
name: "user_features"
description: "User features extracted from event logs"
fabric: "f-small"

features:
...

All the features grouped under this feature set will be computed in a f-small fabric.

Locally

Local fabric config per feature like:

apiVersion: 1
type: featureset
name: "user_features"
description: "User features extracted from event logs"

features:
- name: count_login
fabric: "f-medium"

count_login feature will use a f-medium fabric to be built. Users can override the global fabric config per feature with this method.

Fabrics for models

Fabric config for model training in the model YAML file like:

apiVersion: 1
name: my_model
description: "My model description"

training:
name: my_model_training
fabric: "f-spark-small"