A little over a year ago, I found myself feeling stuck in my role as a data engineer. I had majored in business in college and was looking to connect more with that side of things. I enjoyed my tasks as a data engineer but I wanted more flexibility and creativity. I wanted to be involved in business decisions rather than my tasks already being decided for me. As a data engineer, I felt like I was completing tasks that were already strategically decided on, rather than having a say in prioritization and the needs of the business.
I started applying to different data engineering roles, looking for ones that included more of the day-to-day tasks that I wanted to be involved in like strategic data decisions, warehouse architecture, and automating data models. As you can imagine, it was hard to find a data engineering role that also included business exposure and wasn’t strictly back-end IT development. I started exploring analyst roles as well but found that these didn’t include enough of the technical skills that I had worked so hard to develop the last three years of my career. I felt extremely discouraged, thinking there wasn’t a job out there that I would enjoy yet challenge me in the ways I wanted to grow.
"Then, I stumbled upon a role for an analytics engineer. I had never heard of this role before, but, reading the job description, it sounded exactly like what I was looking for. It included both technical and business exposure. It required collaboration across different teams within a company. I could interact with growth and marketing teams while still coding in SQL and Python and expanding my coding skills."
I applied to that role and have now been working as an analytics engineer for almost a year. I haven’t once looked back!
If you’re in a similar position, not knowing which direction to move in with your data career, I hope you can use this to get a better understanding of the engineering roles out there. Here’s a comparison between my roles as a data engineer and analytics engineer.
What does a data engineer do?
Let’s start off by talking about the role of a data engineer. A data engineer is someone who builds the infrastructure to support the storing and movement of data. They aren’t looking at the data itself but rather focusing on how they can support the data. This includes using Python to write data pipelines, Spark to process data, and cloud technologies like AWS to deploy infrastructure. A data engineer works closely with other developers such as software engineers.
When I worked as a data engineer I first focused solely on DevOps and deploying infrastructure using AWS. I used different pipeline tools such as Data Dog and Jenkins to monitor and deploy code. I wrote a lot of Python scripts that cleaned up our AWS environments and deployed EC2 clusters. However, in my second year, I actually worked more with SQL and dbt. This was untraditional for data engineers who typically work with software development languages and cloud platforms like AWS and Azure. It really depends on the company you work for and how they label their roles. Still, I focused on building applications like Python APIs to help capture our data. Again, I was building infrastructure and processing tools, not looking at the data itself.
Top skills required for data engineering:
Python
AWS
Git
Bash
Spark
Hadoop
The day-to-day of a data engineer involves meeting with a scrum master to help prioritize your tasks and move you along. Instead of meeting with business teams, there is usually some sort of middle person like this to tell you what should be prioritized. Then, you may meet with other data engineers or development teams to decide on how your data processes will be built. This includes design and the specifics of what needs to be deployed. A lot of the day includes heads downtime to really focus on your code and what you are building. Most days I would be writing a Python script to make our deployments quicker or help test infrastructure changes before they were made.
What does an analytics engineer do?
Now, while many might know what it takes to be a data engineer, let’s talk about the skills needed to be an analytics engineer. You can expect to utilize your technical skills, just for a fairly different purpose. You can also expect to utilize more of your business-savy knowledge, using your data to determine the direction of the business.
First, how would you define an analytics engineer? An analytics engineer is someone who moves and transforms data from the source so that it can be easily analyzed, visualized, and acted upon by the data analyst or business user. Analytics engineers deal with the data itself as well as the moving of data. It is their job to make sure data is ingested, transformed, scheduled, and ready to be used for analytics. Many analytics engineers are the brains behind the “modern data stack”. They decide which tools to use for ETL/ELT and then set them up.
Top skills required for analytics engineering:
SQL
Experience with dbt
Communication
Python
Experience with modern data stack tools (Snowflake, Google Big Query, Fivetran, Matillion, Airbyte, Looker, ThoughtSpot, etc.)
Let’s dig into some of these skills more deeply.
SQL and dbt
Just like a data analyst, analytics engineers eat and breathe SQL. We use SQL analytics functions every day to write our transformations within data models. While data & BI analysts may prefer low-code point and click platforms to generate SQL, an analytics engineer starts with SQL for greater control and complex queries. SQL is hands down one of the most important skills that you need to master to become an analytics engineer. With this also comes experience with dbt. Dbt is the leading data transformation tool in the industry and the company that created the role of the analytics engineer! You will most likely be using this to write your data models. Dbt Cloud is pretty easy to learn as long as you have your SQL down since this is the language it utilizes.
Python
While analytics engineers don’t need to be master coders like data or software engineers, it is still important that they know how to code in Python. A majority of data pipeline tools utilize Python and require it when writing your own pipeline. Some may argue that data engineers own data pipelines, but it really depends on the company and bandwidth of your team. I know both data engineers and analytics engineers who own this. Personally, I maintain this for my team because we don’t have any data engineers.
Modern data stack tools
Lastly, an analytics engineer needs to be familiar with the most popular tools in a modern data stack. This means familiarity with ingestion, transformation, warehousing, and deployment tools. You don’t necessarily need to know every tool out there, but you should be familiar with the concepts behind each of them. If you know one tool in each part of the stack, you can probably pick up on the others. Personally, I find this to be one of the most fun parts of the role. There is always a new tool to learn!
The Modern Cloud Data & Analytics Portfolio
Communication
Communication is one of the most important non-technical skills for any role that interacts with business teams. Communication is key as an analytics engineer because it is your job to keep everyone up-to-date on the status of your data. You need to communicate when data quality is compromised, a pipeline is down, and most importantly, to understand what the business needs. Business teams and data analysts are your most important stakeholders. If you don’t properly communicate, then incorrect assumptions can be made on poor-quality data while opportunities get left on the table.
What analytics engineering is not
-
Not building the architecture to support the data.
Don’t expect to be doing work that is similar to software engineering. You won’t be building any applications or infrastructure. You may be using cloud platforms like AWS but you won’t be doing much besides spinning up ECS clusters to support a modern data stack tool you are using.
-
Not your typical software development languages.
You don’t need to have a computer science degree to be an analytics engineer. You should have knowledge of how simple functions such as those written in Python work. You should be able to read basic code but you don’t necessarily need to know how to produce it. While Python is important, you don’t need to have knowledge of JavaScript, React, or Angular. You don’t even need to know Bash.
-
Not only interacting with other engineers and product owners.
You can expect to interact with people that are more business-focused and have little technical knowledge. This means you need to know how to communicate with those that aren’t as technical like you. As an analytics engineer you need to know how to speak business as well as technically. Rather than being told what to prioritize by a product owner, you help shape your team’s priorities. The main stakeholder of an analytics engineer is the data analyst that they work with. Together, you and the analyst decide what parts of the business are top priority.
When to choose one versus the other
So, how do you choose between two important, but different fields? Here are some quick tips.
Analytics engineering may be right for you if:
-
You have a good understanding of the business and you enjoy solving their problems.
To be an analytics engineer you need to understand business terms like ROI, retention, KPI, or in marketing, customer acquisition costs (CAC). You are serving the business in order to help it grow and make money. It can be easy to become disconnected from this fact as a data engineer. However, as an analytics engineer, you need to keep this in mind with all your tasks. You will always be thinking about new ways you can better solve problems using data.
-
You enjoy learning and working with new tools.
As an analytics engineer, you are always looking for new tools and packages to add to your data stack. You will always find areas where you can improve that can be served with a new tool. Make sure you love to research and read through documentation. This is one of the most frustrating but also most rewarding parts of the job. There’s no better feeling than successfully implementing a package and watching it save hours of your team’s time!
-
You enjoy blazing your own path rather than finding solutions to problems assigned to you.
Analytics engineering is all about being pro-active. You won’t sit back and wait for tasks, you need to seek them out. Data quality and monitoring is a huge part of producing reliable results. When you see a check that can be added to prevent problems, you are expected to implement it. For example, a few months ago I noticed a major drop in our data volume. It turned out that this was happening for weeks without anyone realizing. In order to prevent this in the future, I researched a dbt package that I could add to our data models to alert us when the volume was below average.
Data engineering may be right for you if:
-
You enjoy collaborating with other engineers.
If you enjoy working with other engineers and expanding your developer skills, data engineering may be the best choice for you. As a data engineer you have the opportunity to work with and learn from all different levels of engineers. You can quickly expand your technical knowledge by learning from some of the best.
-
You are interested in the infrastructure and code that supports the building of data pipelines.
With data engineering, you are rarely looking at the data itself. You don’t have the same granularity with it such as a data analyst or data scientist. If you love data but don’t necessarily care about the insights you can squeeze out of it, data engineering may be right for you. You will learn skills that can be easily transferred to traditional software engineering roles such as building APIs, deploying code, and supporting platforms.
-
You want to learn to code in new languages and build new projects.
With data engineering, you constantly have the opportunity to learn a new coding language or development tool like React, Javascript, or Golang. There are a ton of different data engineering teams out there that each use their own sets of technologies. On these teams you will be able to learn and implement new languages to help you build real applications. With data engineering you are usually shipping out an actually product whether it is in the form of a pipeline, web platform, or mobile application. Building these types of customer-facing products (whether your customer is another team or someone who buys from your company) can be extremely rewarding.
Choosing the best path for growth
Analytics engineering is still on the rise as compared to data engineering. You will see far less roles available on LinkedIn, but more and more are popping up each day as businesses see their value. There are definitely more roles to choose from in data engineering. You are able to pick ones that fit your unique skillset, since it can be so diverse depending on the technologies a company uses.
This being said, both roles are in very high demand. Every company is starting to realize that data is one of their most valuable assets. According to Glassdoor, the median salaries for analytics engineers and data engineers in the United States are $111,665 and $114,404 respectively (although data engineers in the Bay Area for large tech companies skew higher.) Since the salaries are so similar, don’t let this be a deciding factor in choosing the career that is right for you. If you are currently a BI analyst, analytics engineering is a solid career progression.
Data engineering is for those that enjoy building infrastructure and platforms, working with other engineers, and want to expand their software development skills. Analytics engineering is for those that breathe SQL, enjoy learning new tools, and want to interact with business teams and data analysts. After making my switch from data engineering to analytics engineering, I am confident that I made the best decision for me. This new career path challenges me while also giving me the flexibility to try new things and help the business grow.
Madison Schott is an analytics engineer for Winc, a wine subscription company that makes all its own wines, where she rebuilt its entire modern data stack. She blogs about analytics engineering, data modeling, and data best practices on Medium. She also has her own weekly newsletter on Substack.
