Delta Lake Videos

Watch the latest videos and webinars for the open-source Delta Lake project.

Delta Lake Community Office Hours (2022-03-03)

Join us for the next Delta Lake Community Office Hours and ask us your #DeltaLake questions featuring Florian Valeye, Ivana Pejava, Shixiong Zhu, Denny Lee, and hosted by Vini Jaiswal . Thanks!

Delta Lake Community Office Hours (2022-02-17)

Join us for the next Delta Lake Community Office Hours and ask us your #DeltaLake questions featuring Dominique Brezinski, Gerhard Brueckl, QP Hou, Denny Lee, and hosted by Vini Jaiswal . Thanks!

Delta Lake Community Office Hours (2022-02-03)

Join us for the next Delta Lake Community Office Hours and ask us your #DeltaLake questions. Thanks!

PrestoDB / Delta Lake Community Office Hours (2022-01-25)

Join us for the next Delta Lake Community Office Hours and ask us your #DeltaLake questions. Thanks!

Delta Lake Community Office Hours (2022-01-20)

Join us for the next Delta Lake Community Office Hours and ask us your #DeltaLake questions. Thanks!

Delta Lake Connector for Presto

Delta lake is an open-source project that enables building a lakehouse architecture on top of existing storage systems such as S3, ADLS, GCS, and HDFS. We - the Presto and Delta Lake communities - have come together to make it easier for Presto to leverage the reliability of data lakes by integrating with Delta Lake. In this session, we would like to share the design decisions and internals of the Presto/Delta connector.

Delta Lake Roadmap 2021 H2

We're starting to get feedback from the Delta Lake community on more integrations per the proposed 2021 H2 roadmap. Through this talk, Vini and Denny will provide a recap of the features including Spark 3.1 support, Delta Sharing released in 2021 H1, and what the community asks for the future roadmap for Delta Lake OSS. There are callouts for OPTIMIZE, Apache Heron, and Trino CTAS support as well as the current integration efforts around Apache Flink, PrestoDB, Apache Pulsar, LakeFS, and Nessie. Don't forget the standalone readers and writers and rust lang API optimizations.

Data + AI World Tour: Delta Lake EMEA Panel

Join us for a fun and informative panel discussion about Delta Lake technical scenarios and use cases. Our panelists come from a variety of backgrounds and will each give us their own unique perspective. Gerhard is a Data Engineer and a Delta Lake contributor, Ivana Pejeva is a Data Scientist and Florian Valeye is an Data Engineering Manager and Delta Lake committer. Come listen in and ask questions.

Why Data is Eating the Universe

Why Data is Eating the Universe: The Coming Age of Massive Sky Surveys From Killer Asteroids to Dark Energy: How Apache Spark and Delta Lake can enable the next generation of discoveries in Astronomy

Data Reliability for Data Lakes

Building a modern data lake requires dealing with a lot of complexity: querying historical data + streaming data simultaneously (lambda architecture), validation to ensure data isn't too messy for data science and machine learning, reprocessing to handle failures, and ensuring ACID-compliant data updates. We created the Delta Lake project, open sourced under the Linux Foundation, to relieve data scientists and data engineers from these complex systems problems and instead enable them to focus on extracting value from data. In this talk, we'll dive into these challenges and how ACID transactions solve them. We'll discuss patterns that emerge when you can focus on data quality and the nitty gritty internals of ACID on Spark which enable this focus.

SmartSQL Queries powered by Delta Engine on Lakehouse

As a data analyst have you ever wanted to be able to simply add some Machine Learning capabilities into your SQL Query? Does your database engine require additional laborious steps in order to leverage Python or R functionality for your data scientist to work on? Do you feel that your teams are siloed from each other preventing you from getting the most out of your data? Join us while we work together to build machine learning algorithms into simple functions that our data analysts can use to build smarts into their analytics.

Tutorial: How Delta Lake Supercharges Data Lakes

Delta Lake's transaction log brings high reliability, performance, and ACID compliant transactions to data lakes. But exactly how does it accomplish this? Working through concrete examples, we will take a close look at how the transaction logs are managed and leveraged by Delta to supercharge data lakes.

Generating Surrogate Keys for your Data Lakehouse with Spark SQL and Delta Lake

For this tech chat, we will discuss a popular data warehousing fundamental - surrogate keys. As we had discussed in various other Delta Lake tech talks, the reliability brought to data lakes by Delta Lake has brought a resurgence of many of the data warehousing fundamentals such as Change Data Capture in data lakes. Surrogate keys are unique and lack any business context so they can stand the test of time when joining domain (or dimensional) and fact data. This can be difficult in single-node systems and can be even more complex for distributed systems. In this session, we will discuss the history and value of surrogate keys and what are the requirements for good strategies to implement this data warehousing fundamental into your Delta Lake.

Delta Lake 0.7.0 + Spark 3.0 AMA

On August 18th, 2020, join Apache Spark and Delta Lake committers Burak Yavuz, Tathagata Das, and Denny Lee for an illuminating "Ask Me Anything" session. Whether you would like to know more about the history of Apache Spark to the current bleeding edge use cases of Spark 3.0 and Delta Lake, this is the session to ask your questions!

Simplifying Disaster Recovery with Delta Lake

There's a need to develop a recovery process for Delta table in a DR scenario. Cloud multi-region sync is Asynchronous. This type of replication does not guarantee the chronological order of files at the target (DR) region. In some cases, we can expect large files to arrive later than small files. With Delta Lake, this can create an incomplete version at the DR site at the breakup point. The assumption is that the Primary (Prod) site is not reachable and therefore there's a need to identify and fix the incomplete version of the Delta Lake table. Similar scenarios happen with RDBMS replication, they rely on their logs to restore the database to a stable version and run the recovery or reload process. This document will address this need and look for a solution that can be shared with customers.

Real-Time Forecasting at Scale using Delta Lake and Delta Caching

GumGum receives around 30 billion programmatic inventory impressions amounting to 25 TB of data each day. Inventory impression is the real estate to show potential ads on a publisher page. By generating near-real-time inventory forecast based on campaign-specific targeting rules, GumGum enables the account managers to set up successful future campaigns. This talk will highlight the data pipelines and architecture that help the company achieve a forecast response time of less than 30 seconds for this scale.

Patterns and Operational Insights from the First Users of Delta Lake

Cyber threat detection and response requires demanding work loads over large volumes of log and telemetry data. A few years ago I came to Apple after building such a system at another FAANG company, and my boss asked me to do it again. I learned a lot from my prior experience using Apache Spark and AWS S3 at massive scale some good patterns, but also some bad patterns and pieces of technology that I wanted to avoid.

Machine Learning Data Lineage with MLflow and Delta Lake

In this talk, we will give a detailed introduction to two popular features: MLflow Model Registry and Delta Lake Time Travel, as well as how they can work together to help create a full data lineage in machine learning pipelines.

Best Practices for Building Robust Data Platform with Apache Spark and Delta

This talk will focus on Journey of technical challenges, trade offs and ground-breaking achievements for building performant and scalable pipelines from the experience working with our customers. The problems encountered are shared by many organizations and so the lessons learned and best practices are widely applicable.

Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow

As Atlassian continues to scale to more and more customers, the demand for our legendary support continues to grow. Atlassian needs to maintain balance between the staffing levels needed to service this increasing support ticket volume with the budgetary constraints needed to keep the business healthy – automated ticket volume forecasting is at the centre of this delicate balance.

Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL Pipeline

At Mars Petcare (in a division known as Kinship Data & Analytics) we are building out the Petcare Data Platform – a cloud based Data Lake solution. Leveraging Microsoft Azure, we were faced with important decisions around tools and design. We chose Delta Lake as a storage layer to build out our platform and bring insight to the science community across Mars Petcare.

Parallelization of Structured Streaming Jobs Using Delta Lake

We'll tackle the problem of running streaming jobs from another perspective using Databricks Delta Lake, while examining some of the current issues that we faced at Tubi while running regular structured streaming. A quick overview on why we transitioned from parquet data files to delta and the problems it solved for us in running our streaming jobs. After converting our datasets to Delta Lake.

VIP Ask Me Anything (AMA) Session: Delta Lake

Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms.

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Recently, a set of modern table formats such as Delta Lake, Hudi, Iceberg spring out. Along with Hive Metastore these table formats are trying to solve problems that stand in traditional data lake for a long time with their declared features like ACID, schema evolution, upsert, time travel, incremental consumption etc. This talk will share the research that we did for the comparison about the key features and design these table format holds, the maturity of features, such as APIs expose to end user, how to work with compute engines and finally a comprehensive benchmark about transaction, upsert and mass partitions will be shared as references to audiences.

Operationalizing Big Data Pipelines At Scale

Running a global, world-class business with data-driven decision making requires ingesting and processing diverse sets of data at tremendous scale. How does a company achieve this while ensuring quality and honoring their commitment as responsible stewards of data? This session will detail how Starbucks has embraced big data, building robust, high-quality pipelines for faster insights to drive world-class customer experiences.

Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake

Change Data Capture CDC is a typical use case in Real-Time Data Warehousing. It tracks the data change log -binlog- of a relational database [OLTP], and replay these change log timely to an external storage to do Real-Time OLAP, such as delta/kudu. To implement a robust CDC streaming pipeline, lots of factors should be concerned, such as how to ensure data accuracy , how to process OLTP source schema changed, whether it is easy to build for variety databases with less code. This talk will share the practice for simplify CDC pipeline with SparkStreaming SQL and Delta Lake.

Columbia Migrates from Legacy Data Warehouse to an Open Data Platform with Delta Lake

Columbia is a data-driven enterprise, integrating data from all line-of-business-systems to manage its wholesale and retail businesses. This includes integrating real-time and batch data to better manage purchase orders and generate accurate consumer demand forecasts. It also includes analyzing product reviews to increase customer satisfaction. In this presentation, we'll walk through how we achieved a 70% reduction in pipeline creation time and reduced ETL workload times from four hours with previous data warehouses to minutes using Azure Databricks, hence enabling near real-time analytics. We migrated from multiple legacy data warehouses, run by individual lines of business, to a single scalable, reliable, performant data lake on top of Azure and Delta Lake.

Building Data Quality Audit Framework using Delta Lake at Cerner

Cerner needs to know what assets it owns, where they are located, and the status of those assets. A configuration management system is an inventory of IT assets and IT things like servers, network devices, storage arrays, and software licenses. There was a need to bring all the data sources into one place so that Cerner has a single source of truth for configuration. This gave birth to a data platform called Beacon. Bad data quality has a significant business costs in time, effort and accuracy. Poor-quality data is often pegged as the source of operational snafus, inaccurate analytics and ill-conceived business strategies. In our case since configuration data is largely used in making decisions about security, incident management, cost analysis etc it caused downstream impact due to gaps in data. To handle data quality issues, Databricks and Delta Lake was introduced at the helm of the data pipeline architecture. In this talk we'll describe the journey behind building an end to end pipeline conformed to CI/CD standards of the industry from data ingestion, processing, reporting to machine learning and how Delta Lake plays a vital role in not only catching data issues but make it scalable and re-usable for other teams. We'll talk about the challenges faced in between and lessons learned from it.

Powering Interactive BI Analytics with Presto and Delta Lake

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Comcast, GrubHub, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. Delta Lake, a storage layer originally invented by Databricks and recently open sourced, brings ACID capabilities to big datasets held in Object Storage. While initially designed for Spark, Delta Lake now supports multiple query compute engines. In particular, Starburst, developed a native integration for Presto that leverages Delta-specific performance optimizations. In this talk we show how a combination of Presto, Spark Streaming, and Delta Lake into one architecture supports highly concurrent and interactive BI analytics. Furthermore Presto enables query-time correlations between S3-based IoT data, customer data in a legacy Oracle database, and web log data in Elasticsearch.

VIP Ask Me Anything (AMA) Session: Delta Lake 0.7.0 Early Preview

How Starbucks is Achieving Enterprise Data and ML at Scale

Starbucks makes sure that everything we do is through the lens of humanity - from our commitment to the highest quality coffee in the world, to the way we engage with our customers and communities to do business responsibly. A key aspect to ensuring those world-class customer experiences is data. This talk highlights the Enterprise Data Analytics mission at Starbucks that helps making decisions powered by data at tremendous scale. This includes everything ranging from processing data at petabyte scale with governed processes, deploying platforms at the speed-of-business and enabling ML across the enterprise. In this session, Vish Subramanian will detail how Starbucks has built world-class Enterprise data platforms to drive world-class customer experiences.

Realizing the Vision of the Data Lakehouse

This keynote by Databricks CEO, Ali Ghodsi, explains why the open source Delta Lake project takes the industry closer to realizing the full potential of the data lakehouse, including new capabilities within the Databricks Unified Data Analytics platform to significantly accelerate performance. In addition, Ali will announce new open source capabilities to collaboratively run SQL queries against your data lake, build live dashboards, and alert on important changes to make it easier for all data teams to analyze and understand their data.

Building a Better Delta Lake with Talend and Databricks

With the introduction of Delta Lake last year, a well-tested pattern of building out the bronze, silver, and gold data architecture approach has proven useful. This session will review how to use Talend Data Fabric to accelerate the development of a Delta Lake using highly productive, scalable, and enterprise ready data flow tools. Covered in this section are demonstrations of ingesting 'Bronze' data, refining 'Silver' data tables, and performing Feature Engineering for 'Gold' tables.

Delta from a Data Engineer's Perspective

Take a walk through the daily struggles of a data engineer in this presentation as we cover what is truly needed to create robust end to end Big Data solutions.

Slowly Changing Dimensions (SCD) Type 2

We will discuss a popular online analytics processing (OLAP) fundamental - slowly changing dimensions (SCD) - specifically Type-2. As we have discussed in various other Delta Lake tech talks, the reliability brought to data lakes by Delta Lake has brought a resurgence of many of the data warehousing fundamentals such as Change Data Capture in data lakes. Type 2 SCD within data warehousing allows you to keep track of both the history and current data over time. We will discuss how to apply these concepts to your data lake within the context of the market segmentation of a climbing eCommerce site.

Best Practices on How to Process and Analyze Audit Logs with Delta Lake and Structured Streaming

In this tech conversation, Denny Lee will interview Craig Ng and Miklos Christine to discuss the best practices on how to process and analyze Databricks audit logs using Delta Lake and Structured Streaming. We will discuss and demonstrate how administrators can utilize audit logs to track resource usage and identify these potentially costly anti-patterns.

Data and AI Talk with Databricks Co-Founder, Matei Zaharia

We are happy to have Matei Zaharia join this month's Data and AI Talk Matei Zaharia is an assistant professor at Stanford CS, where he works on computer systems and machine learning as part of Stanford DAWN. He is also co-founder and Chief Technologist of Databricks, the data and AI platform startup. During his Ph.D., Matei started the Apache Spark project, which is now one of the most widely used frameworks for distributed data processing. He also co-started other widely used data and AI software such as MLflow, Apache Mesos and Spark Streaming. Join this great session where we will discuss these technologies as well as answer selected questions posted in the comment section.

Addressing GDPR and CCPA Scenarios with Delta Lake and Apache Spark™

Your organization may manage hundreds of terabytes worth of personal information in your cloud. Bringing these datasets into GDPR and CCPA compliance is of paramount importance, but this can be a big challenge, especially for larger datasets stored in data lakes. Learn how you can use Delta Lake which is created by Databricks and powered by Apache Spark™ to manage GDPR and CCPA compliance for your data lake. Because Delta Lake adds a transactional layer that provides structured data management on top of your data lake, it can dramatically simplify and accelerate your ability to locate and remove personal information (also known as "personal data") in response to consumer GDPR or CCPA requests without disrupting your data pipelines.

Using Delta as a Change Data Capture Source

While it is common to use Delta Lake as a sink for change data captured from traditional data sources; customers are increasingly asking how to use Delta tables as a source for a change data capture (CDC) process. To state a different way, how can we read a stream of changes from a Delta table, so that they can be propagated downstream.

Predictive Maintenance (PdM) on IoT Data for Early Fault Detection w/ Delta Lake

Predictive Maintenance (PdM) is different from other routine or time-based maintenance approaches as it combines various sensor readings and sophisticated analytics on thousands of logged events in near real time and promises several fold improvements in cost savings because tasks are performed only when warranted. The top industries leading the IoT revolution include manufacturing, transportation, utilities, healthcare, consumer electronics & cars. The global market size for this is expected to grow at a CAGR of 28%. PdM plays a key role in Industry 4.0 to help corporations not only reduce unplanned downtimes, but also improve productivity and safety. The collaborative Data and Analytics platform from Databricks is a great technology fit to facilitate these use cases by providing a single unified platform to ingest the sensor data, perform the necessary transformations and exploration, run ML and generate valuable insights.

Diving into Delta Lake Part 3: How do DELETE, UPDATE, and MERGE work

In the earlier Delta Lake Internals tech talk series sessions, we described how the Delta Lake transaction log works. In this session, we will dive deeper into how commits, snapshot isolation, and partition and files change when performing deletes, updates, merges, and structured streaming.

Diving into Delta Lake Part 2: Enforcing and Evolving the Schema

As business problems and requirements evolve over time, so too does the structure of your data. With Delta Lake, as the data changes, incorporating new dimensions is easy. Users have access to simple semantics to control the schema of their tables. These tools include schema enforcement, which prevents users from accidentally polluting their tables with mistakes or garbage data, as well as schema evolution, which enables them to automatically add new columns of rich data when those columns belong. In this webinar, we'll dive into the use of these tools.

Diving into Delta Lake Part 1: Unpacking the Transaction Log

The transaction log is key to understanding Delta Lake because it is the common thread that runs through many of its most important features, including ACID transactions, scalable metadata handling, time travel, and more. In this session, we'll explore what the Delta Lake transaction log is, how it works at the file level, and how it offers an elegant solution to the problem of multiple concurrent reads and writes.

Simplify and Scale Data Engineering Pipelines with Delta Lake

A common data engineering pipeline architecture uses tables that correspond to different quality levels, progressively adding structure to the data: data ingestion ("Bronze" tables), transformation/feature engineering ("Silver" tables), and machine learning training or prediction ("Gold" tables). Combined, we refer to these tables as a "multi-hop" architecture. It allows data engineers to build a pipeline that begins with raw data as a "single source of truth" from which everything flows. In this session, we will show how to build a scalable data engineering data pipeline using Delta Lake.

Beyond Lambda: Introducing Delta Architecture

Online Tech Talk with Denny Lee, Developer Advocate @ Databricks Lambda architecture is a popular technique where records are processed by a batch system and streaming system in parallel. The results are then combined during query time to provide a complete answer.

Getting Data Ready for Data Science with Delta Lake and MLflow

Online Tech Talk with Denny Lee, Developer Advocate @ Databricks One must take a holistic view of the entire data analytics realm when it comes to planning for data science initiatives. Data engineering is a key enabler of data science helping furnish reliable, quality data in a timely fashion.

The Genesis of Delta Lake: An Interview with Burak Yavuz

We're re-igniting the Spark Online Meetup! In this live meetup, Denny Lee (Engineer and Developer Advocate at Databricks) interviews Delta Lake engineer Burak Yavuz.

Reliability and Data Quality for Data Lakes and Apache Spark by Michael Armbrust

Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake offers ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

LoyaltyOne Simplifies and Scales Data & Analytics Pipelines With Delta Lake

Learn how LoyaltyOne uses Data Lake to simplify and scale data and analytics pipelines.

Petabytes, Exabytes, and Beyond Managing Delta Lakes for Interactive Queries at Scale

Data production continues to scale up and the techniques for managing it need to scale too. Building pipelines that can process petabytes per day in turn create data lakes with exabytes of historical data. At Databricks, we help our customers turn these data lakes into gold mines of valuable information using Apache Spark.

Training: Building Reliable Data Lakes at Scale with Delta Lake

Most data practitioners grapple with data reliability issues—it's the bane of their existence. Data engineers, in particular, strive to design, deploy, and serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets.

Designing ETL Pipelines with Structured Streaming and Delta Lake

Structured Streaming has proven to be the best platform for building distributed stream processing applications. Its unified SQL/Dataset/DataFrame APIs and Spark's built-in functions make it easy for developers to express complex computations. Delta Lake, on the other hand, is the best way to store structured data because it is a open-source storage layer that brings

ACID ORC, Iceberg, and Delta Lake

The reality of most large scale data deployments includes storage decoupled from computation, pipelines operating directly on files and metadata services with no locking mechanisms or transaction tracking. For this reason, attempts at achieving transactional behavior, snapshot isolation, safe schema evolution or performant support for CRUD operations has always been marred with tradeoffs.

Building an AI Powered Retail Experience with Delta Lake

Zalando SE is Europe's leading online fashion platform and connects customers, brands and partners. With millions of visitors each month, we have petabytes of purchase, click-stream, product and other data in our data lake. This data is crucial to powering insights on shopper behavior and driving an AI-first strategy to improve site engagement.

Winning the Audience with AI: How Comcast Built An Agile Data And Ai Platform At Scale

Comcast is the largest cable and internet provider in the US, reaching more than 30 million customers. Over the last couple years, Comcast has transformed the customer experience using machine learning. For example, Comcast uses machine learning to power the X1 voice remote, which was used over 8B times in 2018 by our customers to...

Building Data Pipelines Using Structured Streaming and Delta Lake

Given the rise of IoT and other real-time sources and businesses' desire to draw fast insights, there is a growing imperative for data professionals to build streaming data pipelines. Given the plethora of different tools and frameworks in the big data community, it is challenging to architect such pipelines correctly that achieve the desired performance...

Simplify and Scale Data Engineering Pipelines

Delta Architecture, a Step Beyond Lambda Architecture

Lambda architecture is a popular technique where records are processed by a batch system and streaming system in parallel. The results are then combined during query time to provide a complete answer. Strict latency requirements to process old and recently generated events made this architecture popular. The key downside to this architecture is the development...

Making Apache Spark™ Better with Delta Lake

Apache Spark™ is the dominant processing framework for big data. Delta Lake adds reliability to Spark so your analytics and machine learning initiatives have ready access to quality, reliable data. This webinar covers the use of Delta Lake to enhance data reliability for Spark environments. Topics areas include: The role of Apache Spark in big...

Getting Data Ready for Data Science

One must take a holistic view of the entire data analytics realm when it comes to planning for data science initiatives. Data engineering is a key enabler of data science helping furnish reliable, quality data in a timely fashion. Delta Lake, an open source storage layer that brings reliability to data lakes can help take...

Delta Lake - Open Source Reliability for Data Lakes

Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs. Specifically, Delta Lake offers: ACID transactions on Spark...

Announcing Delta Lake Open Source Project | Ali Ghodsi (Databricks), Michael Armbrust (Databricks)

Keynote from Spark + AI Summit 2019

Threat Detection and Response at Scale (Dominque Brezinski & Michael Armbrust)

We approached Databricks with a set of challenges to collaborate on: provide a stable and optimized platform for Unified Analytics that allows our team to focus on value delivery using streaming, SQL, graph, and ML; leverage decoupled storage and compute while delivering high performance over a broad set of workloads; use S3 notifications instead of list operations; remove Hive Metastore from the write path; and approach indexed response times for our more common search cases, without hard-to-scale index maintenance, over our entire retention window. This is about the fruit of that collaboration.