A little over a year ago, I found myself feeling stuck in my role as a data engineer. I had majored in business in college and was looking to connect more with that side of things. I enjoyed my tasks as a data engineer but I wanted more flexibility and creativity. I wanted to be involved in business decisions rather than my tasks already being decided for me. As a data engineer, I felt like I was completing tasks that were already strategically decided on, rather than having a say in prioritization and the needs of the business.
I started applying to different data engineering roles, looking for ones that included more of the day-to-day tasks that I wanted to be involved in like strategic data decisions, warehouse architecture, and automating data models. As you can imagine, it was hard to find a data engineering role that also included business exposure and wasn’t strictly back-end IT development. I started exploring analyst roles as well but found that these didn’t include enough of the technical skills that I had worked so hard to develop the last three years of my career. I felt extremely discouraged, thinking there wasn’t a job out there that I would enjoy yet challenge me in the ways I wanted to grow.
"Then, I stumbled upon a role for an analytics engineer. I had never heard of this role before, but, reading the job description, it sounded exactly like what I was looking for. It included both technical and business exposure. It required collaboration across different teams within a company. I could interact with growth and marketing teams while still coding in SQL and Python and expanding my coding skills."
I applied to that role and have now been working as an analytics engineer for almost a year. I haven’t once looked back!
If you’re in a similar position, not knowing which direction to move in with your data career, I hope you can use this to get a better understanding of the engineering roles out there. Here’s a comparison between my roles as a data engineer and analytics engineer.
Let’s start off by talking about the role of a data engineer. A data engineer is someone who builds the infrastructure to support the storing and movement of data. They aren’t looking at the data itself but rather focusing on how they can support the data. This includes using Python to write data pipelines, Spark to process data, and cloud technologies like AWS to deploy infrastructure. A data engineer works closely with other developers such as software engineers.
When I worked as a data engineer I first focused solely on DevOps and deploying infrastructure using AWS. I used different pipeline tools such as Data Dog and Jenkins to monitor and deploy code. I wrote a lot of Python scripts that cleaned up our AWS environments and deployed EC2 clusters. However, in my second year, I actually worked more with SQL and dbt. This was untraditional for data engineers who typically work with software development languages and cloud platforms like AWS and Azure. It really depends on the company you work for and how they label their roles. Still, I focused on building applications like Python APIs to help capture our data. Again, I was building infrastructure and processing tools, not looking at the data itself.
The day-to-day of a data engineer involves meeting with a scrum master to help prioritize your tasks and move you along. Instead of meeting with business teams, there is usually some sort of middle person like this to tell you what should be prioritized. Then, you may meet with other data engineers or development teams to decide on how your data processes will be built. This includes design and the specifics of what needs to be deployed. A lot of the day includes heads downtime to really focus on your code and what you are building. Most days I would be writing a Python script to make our deployments quicker or help test infrastructure changes before they were made.
Now, while many might know what it takes to be a data engineer, let’s talk about the skills needed to be an analytics engineer. You can expect to utilize your technical skills, just for a fairly different purpose. You can also expect to utilize more of your business-savy knowledge, using your data to determine the direction of the business.
First, how would you define an analytics engineer? An analytics engineer is someone who moves and transforms data from the source so that it can be easily analyzed, visualized, and acted upon by the data analyst or business user. Analytics engineers deal with the data itself as well as the moving of data. It is their job to make sure data is ingested, transformed, scheduled, and ready to be used for analytics. Many analytics engineers are the brains behind the “modern data stack”. They decide which tools to use for ETL/ELT and then set them up.
Experience with dbt
Experience with modern data stack tools (Snowflake, Google Big Query, Fivetran, Matillion, Airbyte, Looker, ThoughtSpot, etc.)
Let’s dig into some of these skills more deeply.
Just like a data analyst, analytics engineers eat and breathe SQL. We use SQL analytics functions every day to write our transformations within data models. While data & BI analysts may prefer low-code point and click platforms to generate SQL, an analytics engineer starts with SQL for greater control and complex queries. SQL is hands down one of the most important skills that you need to master to become an analytics engineer. With this also comes experience with dbt. Dbt is the leading data transformation tool in the industry and the company that created the role of the analytics engineer! You will most likely be using this to write your data models. Dbt Cloud is pretty easy to learn as long as you have your SQL down since this is the language it utilizes.
While analytics engineers don’t need to be master coders like data or software engineers, it is still important that they know how to code in Python. A majority of data pipeline tools utilize Python and require it when writing your own pipeline. Some may argue that data engineers own data pipelines, but it really depends on the company and bandwidth of your team. I know both data engineers and analytics engineers who own this. Personally, I maintain this for my team because we don’t have any data engineers.
Lastly, an analytics engineer needs to be familiar with the most popular tools in a modern data stack. This means familiarity with ingestion, transformation, warehousing, and deployment tools. You don’t necessarily need to know every tool out there, but you should be familiar with the concepts behind each of them. If you know one tool in each part of the stack, you can probably pick up on the others. Personally, I find this to be one of the most fun parts of the role. There is always a new tool to learn!
Communication is one of the most important non-technical skills for any role that interacts with business teams. Communication is key as an analytics engineer because it is your job to keep everyone up-to-date on the status of your data. You need to communicate when data quality is compromised, a pipeline is down, and most importantly, to understand what the business needs. Business teams and data analysts are your most important stakeholders. If you don’t properly communicate, then incorrect assumptions can be made on poor-quality data while opportunities get left on the table.
Don’t expect to be doing work that is similar to software engineering. You won’t be building any applications or infrastructure. You may be using cloud platforms like AWS but you won’t be doing much besides spinning up ECS clusters to support a modern data stack tool you are using.
You can expect to interact with people that are more business-focused and have little technical knowledge. This means you need to know how to communicate with those that aren’t as technical like you. As an analytics engineer you need to know how to speak business as well as technically. Rather than being told what to prioritize by a product owner, you help shape your team’s priorities. The main stakeholder of an analytics engineer is the data analyst that they work with. Together, you and the analyst decide what parts of the business are top priority.
So, how do you choose between two important, but different fields? Here are some quick tips.
Analytics engineering may be right for you if:
To be an analytics engineer you need to understand business terms like ROI, retention, KPI, or in marketing, customer acquisition costs (CAC). You are serving the business in order to help it grow and make money. It can be easy to become disconnected from this fact as a data engineer. However, as an analytics engineer, you need to keep this in mind with all your tasks. You will always be thinking about new ways you can better solve problems using data.
As an analytics engineer, you are always looking for new tools and packages to add to your data stack. You will always find areas where you can improve that can be served with a new tool. Make sure you love to research and read through documentation. This is one of the most frustrating but also most rewarding parts of the job. There’s no better feeling than successfully implementing a package and watching it save hours of your team’s time!
Analytics engineering is all about being pro-active. You won’t sit back and wait for tasks, you need to seek them out. Data quality and monitoring is a huge part of producing reliable results. When you see a check that can be added to prevent problems, you are expected to implement it. For example, a few months ago I noticed a major drop in our data volume. It turned out that this was happening for weeks without anyone realizing. In order to prevent this in the future, I researched a dbt package that I could add to our data models to alert us when the volume was below average.
Data engineering may be right for you if:
If you enjoy working with other engineers and expanding your developer skills, data engineering may be the best choice for you. As a data engineer you have the opportunity to work with and learn from all different levels of engineers. You can quickly expand your technical knowledge by learning from some of the best.
With data engineering, you are rarely looking at the data itself. You don’t have the same granularity with it such as a data analyst or data scientist. If you love data but don’t necessarily care about the insights you can squeeze out of it, data engineering may be right for you. You will learn skills that can be easily transferred to traditional software engineering roles such as building APIs, deploying code, and supporting platforms.
Analytics engineering is still on the rise as compared to data engineering. You will see far less roles available on LinkedIn, but more and more are popping up each day as businesses see their value. There are definitely more roles to choose from in data engineering. You are able to pick ones that fit your unique skillset, since it can be so diverse depending on the technologies a company uses.
This being said, both roles are in very high demand. Every company is starting to realize that data is one of their most valuable assets. According to Glassdoor, the median salaries for analytics engineers and data engineers in the United States are $111,665 and $114,404 respectively (although data engineers in the Bay Area for large tech companies skew higher.) Since the salaries are so similar, don’t let this be a deciding factor in choosing the career that is right for you. If you are currently a BI analyst, analytics engineering is a solid career progression.
Data engineering is for those that enjoy building infrastructure and platforms, working with other engineers, and want to expand their software development skills. Analytics engineering is for those that breathe SQL, enjoy learning new tools, and want to interact with business teams and data analysts. After making my switch from data engineering to analytics engineering, I am confident that I made the best decision for me. This new career path challenges me while also giving me the flexibility to try new things and help the business grow.