Skip to main content

Fabrics

Layer enables you to run your MLOps pipelines effortlessly. These pipelines might require complex environment setups such as specific software and hardware configurations. We call these environments fabrics.

Fabrics enables you to define the following configurations that you need:

  • Hardware: Resource types such as:
    • GPU, CPU, RAM of the machines
    • Single/multiple node clusters
    • Scalability
  • Software: Runtime configs such as:
    • Python runtime version
    • Spark runtime version

Predefined fabrics#

Layer provides predefined fabrics with general purpose resource sets. These fabrics are:

Fabric nameDriver resourceWorker resourceNumber of workersSpark versionPython version
f-smallS23
f-mediumS33
f-spark-xxsmallS13.1.23
f-spark-xsmallS2S223.1.23
f-spark-smallS3S323.1.23
f-spark-mediumS3S343.1.23
f-spark-largeS3S383.1.23

Parameters#

Fabric parameters are defined below.

Fabric parameters are defined below.

Fabric name#

Name of the fabric. You can use this name in the YAML files (dataset.yml, model.yml) to use this fabric as your computation environment.

Driver and worker resources#

Driver and worker resources define the hardware specifications of the fabrics. Layer fabrics have three types of resources:

Resource nameCPUMemory
S111GB
S224GB
S3314GB

Number of workers#

In multi-cluster fabrics, this parameter defines how many worker are needed.

Spark version#

The Spark version of the fabric.

Python Version#

The Python version of the fabric.

How to use Fabrics#

You can pass your choice of fabric, to run their entity computation on. We support fabric configuration in the respective YAML files (model.yml, dataset.yml) in the following format:

fabric: [FABRIC-NAME]

Fabrics for featuresets#

You can use any of the fabrics to build your featuresets in two ways:

Globally#

Global fabric config for all features as a root key in the YML file like:

apiVersion: 1
type: featureset
name: "user_features"
description: "User features extracted from event logs"
fabric: "f-small"
features:
...

All the features grouped under this featureset will be computed in a f-small fabric.

Locally#

Local fabric config per feature like:

apiVersion: 1
type: featureset
name: "user_features"
description: "User features extracted from event logs"
features:
- name: count_login
fabric: "f-medium"

count_login feature will use a f-medium fabric to be built. Users can override the global fabric config per feature with this method.