What is DBT and why so many companies use it?

0:00 [music] Hi, my name is Annie. I'm a tech consultant based in Stockholm. In this tutorial, I will tell you what is DBT and when you should and shouldn't use DBT. Let's get started. During this tutorial, we will cover what is DBT, why it matters, and when to skip it. The DBT is used to transform data already in your warehouse using plain SQL and it comes also with version control testing and documentation. DBT stands for data built tool. So if you look into the data pipeline you can see that first we have sources and those can be your CRM databases or it can be just a data providers providing data.

0:45 Next you have your extract and load tools which can be fit or air tables for example and then after that data is saved in the warehouse and for the warehouse you can use any warehouse it can be snowflake bigquery fabric and so on and and the source data right now is saved raw and messy in your warehouse. Then after that you need to transform the data and then the end is to present this data to stakeholders via report. It can be for example hex streamlid or powerbi reports and dbt comes between warehouse where you have your source raw data and between presentation layer. So you use dbt to transform the data and dbt is only the t in the transformation step. DBT on its own doesn't process your data.

1:35 It only writes different models and those models are just select queries that you write using SQL. And once the models are written then you run DBT models using your warehouse. For example, in Snowflake you use your Snowflake warehouses to run those DBT models. So DBT is the brain. One thing that DBT does really really well is that it is defining the order between tables because when you write DBT in DBT models you use this reference function that is saying ref and then the name of the source that is used. So DBT will write this D or dependency graph for you and it will know exactly which models are dependent on which models. In here for example you can see we have two staging models stage orders and stage customers.

2:26 Then you have one data marked that is all just enrich. And you can see that data marked is getting data from staging models via the ref function and which means that dbt will know for example that every time if there is a change in staging then maybe orders need to be updated or the other way around then also the team knows that there is one contract that is called daily revenue and this table is reading data from orders. So dbt knows about this dependency and when you run dbt run it will respect the dependency. It will first run stages then order and reach then it will run daily revenue. Now let's try to understand what is life without dbt. You will use different store procedures on your data warehouse. You will have SQL files on a share drive or on your computer and then you will run those jobs on its own.

3:16 They will be scheduled and they will run the procedures. So everything works fine and I worked in this setup many many times. But the issue is that if you are new and you don't know what is the order then you are lost. If you're enjoying this tutorial please give me thumbs up and comment DBT so that I continue creating these type of videos. Now back to the tutorial. Another thing is that in nonbt environment you don't have version control. You can't do testing and you can't do documentation. you know that something isn't working when a stakeholder writes you about this. So your actual test is stakeholders noticing the issues and this is not that data people are sloppy. It's just SQL alone has no framework for dependencies tests or documents and when you have the DBT you write your DBT models in your own development environment and they are saved in g and then you can push them to some type of g repository.

4:14 So you have a version control and if you know SQL then you already know 90% of DBT. Testing is built in DBT. You write schema YAML file where for example you are writing the name of the column and then test and then under that you are saying that for example order ID shouldn't be new it should be unique. So you write this information in your schema YAML file and then when you run DBT test all of those tests are run. Same thing when it comes to documentation. So you write short description next to your models and then when you run dbt doc generation DBD builds a searchable site every table every column every linkage from row sources to dashboard and you can see all the dependencies as well in your documentations.

5:02 So DBT isn't a new language. It is software engineering discipline wrapped around the SQL you already know. And should you actually use DBT or not? Let's decide together. the honest decision guide and use DBT. When transformations are SQL in a cloud warehouse, there are more than two people working with the project. Broken data has a real cost and you have outgrown a handful of queries. And do not use DBT if you are still finding out which extract or load tool to use because as we already saw, DBT is for transforming data when the data is already in your data warehouse. Also do not use if you are mainly working with Python, ML and you do not have more than one people in your team or if you are not convenient with Ginga or framework options.

5:50 Want to comment DBT and I will do the next episode where I will show how to install DBT build your first model. That's it for today. Have a good day. Bye.