Milad Madadi
Senior Data Engineer · 10+ years

Milad Madadi

I build data pipelines and lakehouse platforms that people can trust.

I work hands-on in Python, PySpark and SQL. I orchestrate workloads with Apache Airflow, Fabric Data Factory and Azure Data Factory. I've built enterprise warehouses from scratch, led a greenfield Microsoft Fabric platform, and turned slow manual processes into automated, monitored pipelines.

See projects Get in touch
Technologies I use
    Architecture

    From source to consumer

    Watch data move through each layer and change colour as it is refined. Select a stage to see what happens there and the tools behind it.

    The medallion architecture I designed for a UK statutory regulator on Microsoft Fabric, orchestrated with Airflow and Fabric Data Factory.
    Writing

    Blog

    Notes from building and running data platforms: what worked, what broke, and what it cost.

    See all posts on the blog →
    Where I fit

    Projects I can add significant value to

    The kinds of work where my experience shortens the path. Each one is backed by something I've already delivered.

    Moving to Microsoft Fabric or a lakehouse

    Good fit if you're on SQL Server, Synapse or SSRS and want one governed platform.

    Workspace design, OneLake governance, a medallion model from Bronze to Platinum, and a staged migration of existing reports.

    Technologies used

      Cutting cloud data costs

      Good fit if your Azure bill keeps growing or pipelines fail and get re-run.

      Find where pipelines waste compute, rebuild the expensive parts, and remove dependencies on third-party tools.

      Technologies used

        Automating manual reporting

        Good fit if analysts spend days preparing releases or extracts by hand.

        Turn spreadsheet-driven processes into scheduled Python and PySpark pipelines with QA checks built in.

        Technologies used

          Building or consolidating a data warehouse

          Good fit if your data sits in many databases and nobody agrees on the numbers.

          Design the model, replace undocumented stored procedures with clear ETL, and document lineage so others can maintain it.

          Technologies used

            Orchestrating hybrid and legacy pipelines

            Good fit if loads span old and new platforms and run in the wrong order.

            Airflow DAGs for cross-platform dependencies, alongside native Fabric or Azure Data Factory pipelines and proper monitoring.

            Technologies used

              Data quality for regulated or sensitive data

              Good fit if your outputs face regulators, researchers or the public.

              Automated QA and reconciliation, data minimisation, role- and item-level access, and point-in-time history for as-at reporting.

              Technologies used
                Discuss a project
                Selected work

                Projects

                Filter by technology to see where I've used it. You can also click any tag on a card.

                Ingestion & sources Raw & staging Transformation & quality Warehouse & modelling Reporting & publication
                Toolkit

                Skills

                What I use day to day, grouped by where it sits in the data platform.

                2013 – present

                Experience

                1. Mar 2021 – Present

                  Data Engineer

                  UK statutory regulator, United Kingdom
                  • Sole data engineer for a UK statutory regulator. I own the pipelines and platform behind organisational reporting, research datasets and public data publications.
                  • Built the enterprise data warehouse from scratch, replacing undocumented stored procedures with maintainable ETL and clear lineage.
                  • Led the technical design of a greenfield Microsoft Fabric platform with a Delta-based medallion architecture.
                  • Set naming conventions, development standards and source control across all data assets, and created a data wiki covering data dictionaries, pipeline documentation and the asset inventory.
                  • Delivered in weekly Agile sprints, putting the most business-critical datasets first.
                2. Jan 2019 – Aug 2020

                  Lead Data Warehouse & BI Consultant

                  Global group of companies, United Kingdom
                  • Led design and delivery of a central data warehouse for a group of 6,000+ employees across 16 subsidiaries.
                  • Built end-to-end ETL and scheduling frameworks and directed the corporate BI team.
                3. Feb 2017 – Dec 2018

                  Business Intelligence & Data Warehouse Lead

                  Holding group, Iran
                  • Built an enterprise warehouse over 12 SQL Server databases and Excel sources with fully automated SSIS ETL.
                  • Set up the BI function from scratch, recruiting and training two specialists.
                4. Mar 2015 – Jan 2017

                  Senior Business Intelligence Developer

                  Iran
                  • Built ETL in DTS and SSIS across SQL Server, Oracle and Excel, and tuned complex T-SQL for performance and consistency.
                5. Mar 2013 – Dec 2014

                  Business Intelligence Developer

                  Iran
                  • Built ETL from SQL Server, text and Excel sources using QlikView scripting and QVD data models.
                Certifications
                • Implement data engineering solutions using Microsoft FabricMicrosoft
                • Azure Data Engineer AssociateMicrosoft
                • Power BI Data AnalystMicrosoft
                • ICAgile Certified Professional (ICP)ICAgile
                • Introduction to Big Data, UC San Diego2022
                • Data Analysis Using Python, University of Pennsylvania2022
                • Data Warehousing Specialisation, University of Colorado2021
                Education
                • Business & Management, Aston University, UK2021
                • MSc Information Systems Design, University of Tehran2018
                • BSc Computer & Electrical Engineering, Shahid Beheshti University2014
                Contact

                Let's talk about your data platform

                I'm open to senior data engineering roles and projects in the UK, including work that needs SC clearance.

                Milad.Madadi@gmail.com Open mail app