Maokai Lin – 林懋恺

Journals of a curious learner

© All rights reserved.

Maokai Lin – 林懋恺

Journals of a curious learner

© All rights reserved.

Tag: serverless

Engineering

A Serverless Setup with Airflow for Machine Learning

We need to run several thousand of machine learning tasks in production every couple of hours, and the demand is very bursty — many at the beginning of each hour. Combining Airflow with serverless (we use AWS fargate), we manage to run these tasks in a reliable and scalable fashion, with well-controlled costs. This post is a brief introduction of how we set it up, and how different components couple together.
Read More