When working with DynamoDB, I’m pretty sure you already faced the problem of needing to create some kind of parser between dynamoDB object and python dictionary, since they are kind of different, mostly when working with big nested objects or even multiple different schemas.

Godess Athena

First of all, I want to apologize myself for not posting here for a long time. Recently I moved from Social Miner to iFood and I've been working hard to learn everything as fast and better as I can from them.

Before moving to iFood, I had to do a…

Hi guys! Today I’ll make a different kind of post, it will not be a tip as usual but something I built.

I participated in this project which was a layer over Athena that build the query for the user using drag-and-drop features. The application would have many users and…

Creating a single dag dynamically, for example when you need a number of similar tasks to be ran at the same time, is simple. Just use any loop, create tasks, set their upstreams and voilà: the dag is ready.
Obs.: if you are not familiar with this, check this article.

Sometimes when using Pandas it's difficult to manipulate data the way we would do in a relational database. Some simple operations like joins or composite filters are not very intuitive or too painful to execute with Pandas library.

When developing Spark jobs, regardless of language, we may face a problem or situation that requires us to use an specific java library and this can be tough if we do not have an automatic deploy pipeline for the Hadoop cluster or if this pipeline is not configurable.

Spark has…

It’s common to face import errors when creating your own custom plugins and dags while developing for Airflow, mainly because when the Airflow webserver starts, it creates references for the plugins within the plugins folder to its own python path.

Import errors when developing the DAG script
Import errors when developing the DAG script

To solve this problem, I built a simple shell script…

Flávio Teixeira

Data engineer, gamer and addicted to technology. Currently working at iFood (https://institucional.ifood.com.br/)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store