Building a cloud-based log-transfer pipeline (ETL)

Background

Large systems generate logs in many places every day. Logs are essential for investigating incidents and understanding usage. Reliably collecting them and delivering them where they are needed requires a transfer mechanism that doesn't drop or stall data along the way.

The processes also have to be built to fit the cloud environment in use.

What we did

RG joined the project as part of the client's development team. Our responsibility was building the ETL (extract, transform, load) for log transfer in the cloud.

Following the team's policies and the existing architecture, we built the processes that ingest logs, shape them into the required format, and transfer them.

Result

We joined the development team in 2022 and were responsible for building the log-transfer ETL. We have applied that experience to our cloud and data platform work since then.