Homepage
Sign inGet started

Towards Data Engineering

Navigating the Path to Data Engineering Excellence

  • About
  • Trending
    Fail Fast or Quarantine? Two Data Quality Patterns Every Spark Engineer Should Know

    Fail Fast or Quarantine? Two Data Quality Patterns Every Spark Engineer Should Know

    Learn when to fail fast or quarantine bad data in Spark pipelines.
    Go to the profile of Marcel Kennert
    Marcel Kennert
    May 12
    15 Common Spark Errors in the Big Data Industry — Causes, Detection & Detailed Fixes

    15 Common Spark Errors in the Big Data Industry — Causes, Detection & Detailed Fixes

    Apache Spark is widely used for building distributed data processing pipelines, but it frequently encounters operational and runtime…
    Go to the profile of Solon Das
    Solon Das
    May 11
    APACHE ICEBERG — DEEP DIVE II: HOW READ & WRITE OPERATIONS WORKS

    APACHE ICEBERG — DEEP DIVE II: HOW READ & WRITE OPERATIONS WORKS

    As a direct continuation, which I discussed architecture of Iceberg layers i want to close this topic with writing and reading operations.
    Go to the profile of Douglas Souza
    Douglas Souza
    Dec 30, 2024
    Latest
    Say Goodbye to Dirty Data: Build Trustworthy Pipelines with These Pro Tips

    Say Goodbye to Dirty Data: Build Trustworthy Pipelines with These Pro Tips

    Non-members can access the full article through this Link.
    Go to the profile of Ritam Mukherjee
    Ritam Mukherjee
    May 21
    Accelerate your Analytics Journey from On-Prem to Databricks : Introducing Prism

    Accelerate your Analytics Journey from On-Prem to Databricks : Introducing Prism

    A new lens on SAP data — clear, intelligent, real-time
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 20
    100 Days of Data Engineering with Databricks Day 86: Summarizing Product Reviews Using GenAI and…

    100 Days of Data Engineering with Databricks Day 86: Summarizing Product Reviews Using GenAI and…

    So far in our SmartRetail 360+ journey, we’ve analyzed structured data from SAP to build ML models for churn, CLV, recommendations, and…
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 20
    100 Days of Data Engineering with Databricks Day 85: Introducing Generative AI in Retail Pipelines

    100 Days of Data Engineering with Databricks Day 85: Introducing Generative AI in Retail Pipelines

    Powered by Databricks ai_query()
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 20
    100 Days of Data Engineering Day 83: Calculating Customer Lifetime Value (CLV) Using SAP Customer…

    100 Days of Data Engineering Day 83: Calculating Customer Lifetime Value (CLV) Using SAP Customer…

    In the previous article (Day 82), we built a churn prediction model to identify which customers were at risk of becoming inactive using…
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 19
    SQL: JOINS — BASIC CONCEPTS

    SQL: JOINS — BASIC CONCEPTS

    Understand the basic concepts of JOINS with SQL.
    Go to the profile of Douglas Souza
    Douglas Souza
    May 19
    Databricks Data Engineering Day 82: Predicting Churn from SAP Order Trends

    Databricks Data Engineering Day 82: Predicting Churn from SAP Order Trends

    We’ve reached one of the most high-impact use cases in the world of retail analytics: Customer Churn Prediction.
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 19

    How Data Engineering Powers AI-Based Analytics: Feeding the Intelligence Behind the Magic

    Introduction
    Go to the profile of Naidu Rongali - Senior Big Data and ML Engineer
    Naidu Rongali - Senior Big Data and ML Engineer
    May 18
    WealthVista AI Day 1: Why Wealth Management Needs AI-Driven Intelligence Platforms

    WealthVista AI Day 1: Why Wealth Management Needs AI-Driven Intelligence Platforms

    The Shifting Landscape of Wealth Management
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 17
    Databricks Industry Solutions: Unlocking Industry Innovation with 5 Plug-and-Play Data & AI…

    Databricks Industry Solutions: Unlocking Industry Innovation with 5 Plug-and-Play Data & AI…

    Databricks is revolutionizing how industries unlock value from their data. At the forefront of this transformation, we are proud to…
    Go to the profile of THE BRICK LEARNING
    THE BRICK LEARNING
    May 17
    About Towards Data EngineeringLatest StoriesArchiveAbout MediumTermsPrivacyTeams