Mage (still The Move) For Data Pipelines

Mage is an open-source data pipeline tool that lets you build DAGs (directed acyclic graphs, meaning sequences of data transformations) by combining Python or SQL blocks. v1 of Internet Menace recommended Mage because it was approachable, free, and rendered pipelines as actual Python files (not opaque drag-and-drop binaries).

Both reasons still hold, and there's a new one: Mage pipelines pair beautifully with coding agents.

The workflow:

1. You describe a data transformation in natural language to your agent. "Read the orders CSV, filter to the last 30 days, group by customer, compute total revenue per customer, write to a new table."

2. The agent writes the Python (using pandas or polars) for each step.

3. You drop those Python blocks into a Mage DAG.

4. Mage runs the DAG on a schedule. It handles retries, logs, and monitoring.

5. Because the DAG renders as plain Python files, you can have the agent read the rendered code back and check it. "Here's my pipeline. Are the edge cases handled? Is there anything I'm missing?"

This is the loop: agent writes the steps, Mage runs them on a schedule, agent reviews the result. You're orchestrating, not coding.

Install: pip install mage-ai and then mage start project_name. Open the browser to localhost:6789. Build the DAG visually, edit the Python in the blocks, run the pipeline.

Recommendation unchanged from v1: if you're doing any kind of recurring data work, learn Mage. The 90% case is "extract this data from somewhere, clean it up, put it somewhere else, on a schedule." Mage handles all of that without you writing infrastructure code.