Maokai Lin – 林懋恺

Journals of a curious learner

© All rights reserved.

Maokai Lin – 林懋恺

Journals of a curious learner

© All rights reserved.

Tag: backend

Engineering

A Serverless Setup with Airflow for Machine Learning

We need to run several thousand of machine learning tasks in production every couple of hours, and the demand is very bursty — many at the beginning of each hour. Combining Airflow with serverless (we use AWS fargate), we manage to run these tasks in a reliable and scalable fashion, with well-controlled costs. This post is a brief introduction of how we set it up, and how different components couple together.
Read More
Engineering

An Interesting Case about Micro-Service Dependency

Recently we've been working to shard our databases in order to put each client's data into its own database. This requires a new micro-service that provides database connection parameters. This post is about a discussion Steve and I had about its schema design and dependency considerations.
Read More