Machine learning (ML) is complicated. Some pain points are common, though, and Layer is designed with them in mind. This page describes some of the pain points and how Layer helps reduce the pain.
Layer addresses common problems in the way data scientists work.
There is no way of having an end-to-end pipeline from experimentation to production for data scientists without being forced to give up on their current tool-kit, execution environment, programming language, or IDE.
Pain point. There is no standard way-of-working (ML pipeline) defined in data science teams. Most data scientists lack the time or software development skills to build one.
Abstraction. Layer provides seamless in-Code integration, which you control using the Layer SDK.
Orchestration. Layer provides the pipeline. It was designed for data analysts, which is why in Layer, datasets, features, and ML models are first-class entities, not afterthoughts.
Layer provides a platform and tools to address common communication difficulties in the ML space.
Communication problems within data science teams are common: silos, no pair-review, challenging new hire onboarding process, and painful handover of projects when a team member leaves.
With the current standards and way of working, data scientists often don't have a chance to review someone's model details, such as what combination of parameters have been used while optimizing the model.
Single source of truth. Layer is your single source of truth. It acts like an orchestrator behind the scenes to enable data scientists to track end-to-end journeys of projects by exposing data lineage, feature descriptions and codes, model signatures, and so on.
Versioning. Layer gives you automatic versioning. With versioning and a single source of truth, team members can see their teammates' model development iterations with the help of the hyper-parameter tuning and can review them.
Layer removes barriers that prevent collaboration.
Teams waste a lot of time on duplicate work because the tools do not facilitate team collaboration. Current tool sets often do not allow sharing or reuse of common features.
Reusability. Layer provides reusability in its core features.. Data and model catalogs provide central repositories for all features and ML models so that the whole team can have benefit. Keep everything in one place, where everyone can see and use it.
Layer facilitates development and reduces the mental load on team members.
A typical ML model development process usually consists of many experiments, so data scientists need to change their focus often. It is hard to follow and switch between these experiments, leading to stress and confusion.
While optimizing an ML model, most data scientists must write down performance numbers and metrics manually and no other teammate can have a chance to see their work.
Visibility Layer has hyper-parameter tuning to help you tracking model development, and what you see, your team can see.
Reproducability. With Layer's automatic versioning, it's easy to go back to the previous version and reproduce an old experiment with the same results even if changes are made on entities over time.
Tired of waiting for an engineer to find time for you? Layer can help.
The biggest challenge of data science teams is being able to serve their models into production. They are heavily dependent on software engineering teams and thus, do not control their own models.
It's hard to observe models running in production, so you can't trace model performance in real-time.
Even if an ML model is deployed, it is not easy to make a change such as adding a new feature or another model parameter and see its effect on the performance in production right away.
It's hard to do a sanity check on input (streaming) data. Many teams aren't able to monitor if something goes wrong in the data flow.
Deployment tools. It's easy to deploy models with the Layer pipeline. Layer enables data scientists to deploy models of their own to make real-time predictions by providing a REST API endpoint.
Continuous delivery. Layer provides continuous delivery and deployment, enabling the creation of automated and reactive pipelines.
Observability. Layer provides performance monitoring and checks data quality by running automated tests on the data. All processes are traceable, leading to full observability and the ability to monitor models in-production.