From Data Pipelines to Intelligent Performance: Aligning Strategy, Architecture, and AI

Featured Speakers:-

Links and Resources

• The Future of Data: An Augmented Analytics Guide to Next-Gen Decision-Making
• The Ultimate Guide to Data Integration
• Data Governance Framework: Strategies for Reliable Data Management

Transcript

[00:14] Host (Rajnish Ranjan)

Hello, ladies and gentlemen. Welcome to the AI Machines and Marketer Show.

Today, we are having an exciting fireside chat on the topic, “From Data Pipelines to Intelligent Performance: Aligning Strategy, Architecture, and AI.”

Well, today’s session will explore how modern organizations are rethinking their data foundations to support analytics, dashboards, AI automation, and real-time decision-making.

As we know, businesses are increasingly adopting AI- and LLM-driven use cases. The quality, structure, and trustworthiness of data have become more important than ever.

Through our conversation, we hope to understand what separates a basic data platform from a true intelligence platform. We will also learn what our teams should focus on before designing a future-ready data platform.

Without further ado, please welcome our guest speaker, Mr. Rahul Sachdeva, Senior Director of Analytics at Growth Natives, who will share his important insights on today’s session topic.

Hello, Rahul. Welcome to our chat show. So, as my first question, how are you feeling today?

[01:25] Rahul Sachdeva

Well, good to be back here.

[01:28] Host (Rajnish Ranjan)

Definitely. So, Rahul, are you ready for our chat show?

[01:32] Rahul Sachdeva

Let’s hope so.

[01:34] Host (Rajnish Ranjan)

Okay then, let’s begin.

So Rahul, my first question to you: Why is traditional ETL no longer enough, and how is it evolving in modern data platform engineering?

[01:49] Rahul Sachdeva

Yeah, I mean, that’s rather a deep question, I would say.

Legacy ETL, if I talk about it, was standardly made for reporting, which was more around historical data and questions like, “How did we do last quarter?”

The kind of ETL we are dealing with right now, or the kind of use cases we have right now, is more around AI, more around enabling decisions.

So the shift has happened from looking back to looking forward. And consider this.

Let’s say there’s a person who clicks an ad on day one, and then she goes on to fill out a form. Maybe a salesperson calls her, so she goes into the CRM system.

So she has traveled through a few systems here. To her, it is just a single journey, but for our systems, these are three different systems.

We’re starting off with Google Ads. Then you go to marketing automation, become a lead, and then a salesperson calls you up. Then you become a valid opportunity, right?

And let’s say she made the purchase. So it was done and dusted for her in three days. But if we look at our data, we have got five different versions of the same person.

And we have to match that across systems. While she was going through all the decision-making, we have to analyze and see how we can really push or how we can get her to buy the product.

So that is really how the products or the systems are talking to each other in real time, and that’s where ETL is also required.

It’s kind of a mix of operations, AI, and analytics. It’s an interesting era because everything is a whole lot of processes that have been combined together.

Yeah, it’s just converging—all the AI, analytics, and operations together.

[04:15] Host (Rajnish Ranjan)

Right. Yeah, it’s changing a lot, I would say. Definitely.

So Rahul, when we talk about progress, how has data architecture progressed from traditional warehouses to lakehouses and intelligence platforms?

[04:30] Rahul Sachdeva

Yeah, absolutely.

I mean, if you think about the previous systems, those had kind of flat tables, which are more like RDBMS-style.

You have got your own dimension tables, you have got measure tables, and then for reporting purposes, we used to have redundant data in the tables so that we don’t have to join again and again, right?

But now, people are asking these questions to a chatbot, right?

And a chatbot doesn’t care if we are giving it redundant data. It ideally works better on non-redundant data, so normalized data. But the governing layer is the semantic layer.

Sorry, what was the question again?

[05:27] Host (Rajnish Ranjan)

It was like, how have we progressed from traditional warehouses?

[05:31] Rahul Sachdeva

Okay, so yeah, going back to the warehouse question, we used warehouses. We had Postgres, we used different versions.

It’s still used. We still do.

It’s just that those are not supporting the current use cases where we have AI and LLM chatbots querying our data, and that’s where we’re headed.

So we might not be requiring those redundant tables anymore. We might be requiring a semantic layer, which defines how the data is structured, which defines how the data is joined together.

What are the KPIs? What are the definitions?

Those things might reside in the semantic layer, and that’s how the querying or warehousing or reporting would happen, actually.

And then it couples with other open formats like Parquet files, which are read by Python and SQL equally. So those kinds of formats.

ML is another layer. I mean, for ML, we definitely need a different kind of schema.

For example, a feature store. We also do drift monitoring on that, so there are different use cases.

If we talk about ML, we would probably need different feature stores. We would need all the data in one place, and then couple that with drift monitoring.

But if we talk about AI, it’s better to use normalized schemas.

Traditional BI, obviously, or traditional warehouses would rather still be using the redundant tables.

[07:19] Host (Rajnish Ranjan)

So, as we talk about growth and progress, what architecture, ownership, and design foundations are essential for building a scalable and cost-effective data system?

[07:34] Rahul Sachdeva

Right, yeah. Okay, so I would say what we should not do.

[07:42] Host (Rajnish Ranjan)

Okay then, let’s go the other way around. That’s more effective.

[07:48] Rahul Sachdeva

So the data should not be built ad hoc. The data ETL should not be built ad hoc.

And most of the clients we work with definitely start off with something small.

Well, let’s do this kind of reporting. Let’s build these types of things to support our use cases. That is all fine.

But I’m saying when we grow, we should not grow ad hoc. We should take a step back and identify what the total number of use cases is, what the total data sources are, and how we are going to define the schema.

And ideally, there should be a data product rather than having some data pipelines here and there.

So a data product, a data product owner who should be able to define it, and we should be able to get requirements from stakeholders.

There should be SLAs in place. If a stakeholder wants a new field added, when will that be done? And so on and so forth.

So it depends on how the data is structured.

Again, smaller clients might still opt for data pipelines and coding, but as we scale, it is very important that we use a proper data product and establish that as a system.

[09:21] Host (Rajnish Ranjan)

So how do pipeline modeling and processing choices affect the performance and reliability of BI dashboards?

[09:28] Rahul Sachdeva

We should be defining what our data feeds are, what our tables are, what the refresh rate is going to be, and who our customers are.

Do they really need batch, or can they do it once a day, or do they really need real time?

So those kinds of decisions definitely impact the scale and the cost of the system.

And whenever we talk about scalability and reliability, I would also like to build self-healing pipelines.

So that means if there is any failure, you already have an alert, and maybe agentic AI is already working to fix that error in the ETL pipeline.

And that kind of saves us from the embarrassment when the stakeholder calls you up and says, “It’s not fresh,” or “It’s not good,” right?

[10:35] Host (Rajnish Ranjan)

Okay, so when we talk about reliability, what changes in pipeline design, data models, and schemas when an organization introduces AI and LLM use cases?

[10:49] Rahul Sachdeva

Yeah, absolutely.

I mean, since we are already talking about AI and LLMs, I would take this example.

Let’s say we are in a restaurant and there’s already a prepared menu because you can order dishes out of that.

But let’s say you are given an ingredient list, and you can just order any dish, but the recipe has to be defined first.

So you can order any dish in the world as long as there is a standard recipe for it, right?

So even if that wasn’t on the list or the menu, you can still order it.

And that’s exactly how AI is going to change ETL and the reporting part as well.

Earlier, we used to build dashboards for what had happened already. So we already had defined what the question was.

But since we’re using AI and LLMs now, we don’t have to define the question beforehand.

A question is being invented now. A question like, “Tell me the last campaign’s performance,” or “Tell me the enterprise revenue from LinkedIn Ads and compare that with paid search, except this one campaign that skewed the most.”

So you can’t answer that question with the help of a dashboard. You really need something dynamic there.

And those kinds of use cases actually come up, and we have to have a semantic layer as well on top.

So we have the data. We have a semantic layer where we define the rules, where we define what the definitions of those KPIs are and how the data will be joined.

Yeah, it works best like this if we already have a semantic layer. That’s the whole crux of it.

If I have to define what the goal is when we are talking about ETL and AI, I would say context and, obviously, the semantic layer.

[13:05] Host (Rajnish Ranjan)

So Rahul, as we talk about AI and LLMs, one question pops into mind.

Where do data quality and governance challenges typically begin? And how can teams address them within the pipeline?

[13:19] Rahul Sachdeva

Yeah, so I mean, it comes with maturity, first of all, the governance and monitoring part.

But I would say start small.

So even if you have small pipelines, have an alert system, a system like Grafana or Elasticsearch that can trigger alerts whenever there’s a failure of the pipeline.

There are other tools like Airflow, again, which can send out alerts on Slack.

So yeah, there are plenty of tools right now that we could use to set up alerts, and they are across the system, whatever kind of pipeline we use.

But if we talk about governance, that is a bit more challenging.

Because if you’re giving access based on a person’s email address, you’re going to get lost there, right?

Because there would be so many roles, so many accesses and permissions that it is virtually not possible to manage.

So you have to define a role.

And when you assign roles to users, you can apply permissions universally to the role. So you can get access to that particular role, and then that applies to the user.

There are again tools like Immuta, which categorize and define access.

So there are defined stakeholders, defined owners of the data, and then you request access to them through the system, and they provide you access for a certain period of time.

It’s not that you’re on standard access all the time, right?

So yeah, to have these two things in place is essential so that your data does not go into the wrong hands.

[15:27] Host (Rajnish Ranjan)

So that brings me to the last question of our first segment.

What will define the next generation of data platforms as analytics, AI, automation, and real-time decision-making come together?

[15:40] Rahul Sachdeva

Yeah, that’s a big question, but I’ll try to sort it out.

For the last 20 years or so, we designed dashboards and ETL systems for humans, right?

So the next era, or starting now, I think we are building it for agents.

And if you think about all the data that we are collecting, we’re going to feed it to machines, we’re going to feed it to agents, and we’re going to completely automate the journey of a customer.

Serving them ads, marking them as leads, marking them as churn risks, whatever it is.

But there is a big challenge and a big risk in this as well.

Now, as humans, we can sense that there is something wrong, okay?

Machines, they’ll just trust our stuff. And we’ve all heard “garbage in, garbage out,” right?

So it’ll be beautiful garbage out because AI produces beautiful outputs, let’s say. So that will be more refined, but it’ll be garbage.

[16:50] Host (Rajnish Ranjan)

Definitely.

Okay, now that brings me to our second segment, the rapid-fire round.

Are you ready?

[16:58] Rahul Sachdeva

I think so, yeah.

[17:03] Host (Rajnish Ranjan)

My first question to you, Rahul: What is one data architecture trend you believe is overhyped?

[17:09] Rahul Sachdeva

Real time. Everything real time.

It’s not required. I mean, the data should be fresh, not instant.

[17:17] Host (Rajnish Ranjan)

What is the biggest warning sign of an unhealthy data pipeline?

[17:23] Rahul Sachdeva

I’ll say if you cannot tell whether the number is right or wrong, or even worse, if a couple of people cannot agree whether the number is right or wrong.

[17:34] Host (Rajnish Ranjan)

Okay. Should every organization move to a lakehouse architecture?

[17:38] Rahul Sachdeva

No, definitely not. Only wherever it is required.

[17:41] Host (Rajnish Ranjan)

Okay. What matters more, faster data or more trusted data?

[17:45] Rahul Sachdeva

Definitely trusted data.

Faster and wrong makes it garbage.

[17:51] Host (Rajnish Ranjan)

Okay. Can AI compensate for poor data quality?

[17:56] Rahul Sachdeva

Definitely not. It’ll amplify it.

[17:59] Host (Rajnish Ranjan)

Okay. What is the most overlooked driver of cloud data cost?

[18:04] Rahul Sachdeva

I would say scalability.

[18:10] Host (Rajnish Ranjan)

Makes sense.

Who should ultimately own data quality, business or engineering?

[18:15] Rahul Sachdeva

Again, I would say the data owner. The data product owner would be the ideal candidate.

[18:21] Host (Rajnish Ranjan)

Will dashboards remain relevant in an AI-first organization?

[18:25] Rahul Sachdeva

I think yes, but more like an anchor or more like a watchtower rather than a go-to tool.

[18:34] Host (Rajnish Ranjan)

Okay. The last question, in one line: What turns a data platform into an intelligent platform?

[18:41] Rahul Sachdeva

Yeah, so if you’re not looking back into the data and you’re looking forward and you’re basing your decisions on it, I think you already have an intelligence platform.

[18:52] Host (Rajnish Ranjan)

Okay then. Well, that brings us to the end of our chat show today.

Thank you so much for joining us, Rahul.

[18:58] Rahul Sachdeva

Thank you so much.

[18:59] Host (Rajnish Ranjan)

Well, that concludes today’s session, our fireside chat show.

And with today’s conversation, one thing is very clear: strong data pipelines, thoughtful architecture, and trusted data foundations are essential for building intelligent AI-driven platforms.

Thank you, everyone, for joining us today.

We hope this session gives you useful ideas to take forward.

See you in the next episode.

Until then, keep learning, keep evolving.

See more webinar