0:00

Each phase should have a deliverable.

0:02

It might not be a specific

0:04

product or anything, is okay to be

0:06

imperfect, but let's have a goal. And

0:09

when we have a goal and when we have a plan

0:12

at each phase, then

0:14

big foundational work

0:16

might seem achievable. although

0:18

you cannot see an ROI at 1

0:20

and 2, but since I know I

0:23

can give you that ROI back, at

0:25

phase six, so that this is very important.

0:27

In that case, you'll be more convinced than

0:30

me going to you and saying that, Hey, I don't

0:32

really know how to go to phase six, but we need to

0:34

do phase one.

0:36

How can you drive developer productivity,

0:38

lower costs and deliver better

0:40

products? On June 20th and

0:42

27th LinearB is hosting a workshop

0:45

that explores the software engineering intelligence

0:47

category. We'll showcase how data-driven

0:49

insights and innovative workflow automations

0:51

can optimize your software delivery practices.

0:54

At the end of the workshop, you'll leave with a complimentary

0:57

Gartner market guide to software

0:59

engineering, intelligence, and other resources to

1:01

help you get started. Head to the show notes

1:03

or LinearB dot IO slash events

1:05

to sign up today.

1:06

Welcome back to Dev Interrupted, I'm your co host Conor

1:09

Bronsdon. Today I'm being joined by Sanghamitra Goswami,

1:12

Senior Director of Data Science and Machine Learning

1:14

at PagerDuty. Sanghamitra,

1:16

thank you so much for joining me.

1:18

Thank you, Conor, for inviting me.

1:20

Honestly, it's my pleasure. I think we

1:22

could really benefit from your expertise as

1:24

someone who has such a deep understanding

1:27

of the approach that data scientists have taken

1:29

to developing AI models. And,

1:31

I mean, frankly, AI is all the rage right now, right?

1:33

Everyone's talking about it. Everyone has opinions

1:36

on what it is. And so it's important that

1:38

we level set with the audience a bit, and

1:40

have the opportunity here to pick the brain of a

1:43

data science leader like you, and understand

1:45

how engineering teams can

1:47

translate this and leverage

1:49

AI models, or LLMs in

1:52

particular, within their org. Let's

1:55

maybe unravel some of the strategies

1:57

that champion the role of data science,

1:59

machine learning, and AI teams, and help our

2:01

audience understand how to navigate this

2:04

emerging future and ever expanding landscape.

2:06

Why don't we talk a bit about the history of LLMs,

2:09

what they are. and why they're so important.

2:12

You know, Conor, it's been a crazy time

2:14

now. Um, nine months back

2:16

when ChildGBD came out, uh,

2:18

PagerDuty leadership, they told me, Mitra, we need

2:20

to do something with, uh, LLMs, and it,

2:23

it's crazy the way the world is saying

2:25

that, hey, we, we want to do LLM, let's

2:27

have a feature that uses

2:29

LLMs. So what are LLMs?

2:32

Large language models, they leverage

2:34

foundational machine learning, AI, deep

2:36

learning. Models to understand

2:38

natural language and to give answers

2:41

as so they can talk to us like a bot

2:44

You look at the history of NLP, it's based on

2:46

NLP, it's based on the NLP models

2:48

that we have built out. If you look at the history

2:50

of NLP, in 1948, Shannon

2:52

and Weaver, the first paper came out.

2:55

And, and, during that time,

2:57

we didn't have the computer storage that is

2:59

possible now. We didn't, we couldn't

3:01

really have a lot of computational

3:04

power. So, It

3:06

was not possible to always run

3:09

these large language models because they require

3:11

large amounts of

3:13

text. Right. However,

3:15

from that start, where Shannon

3:18

and we were in 1948, if you fast forward,

3:21

um, I would actually mention

3:23

there is another milestone

3:25

where the transformer architecture got

3:27

introduced and, um,

3:30

attention is all you need is the paper. So,

3:32

if I look at these two milestones and how

3:34

the landscape has changed, the

3:36

computational power, GPUs.

3:39

So with everything in mind, now is the

3:41

perfect time that we can all

3:43

reap the benefits of years

3:45

of research and computational

3:48

power. And engineering in that

3:50

sense. So this is the perfect time.

3:52

Absolutely. It's, it's really interesting to think about

3:54

how, even just a few years ago before we realized

3:57

that we could paralyze processes within

3:59

AI model development, uh, using

4:01

GPUs, there just wasn't

4:03

this speed of development that we've seen around AI models

4:05

today. I know. Uh, so I, I'd

4:08

love to kind of talk about how

4:11

teams can actionably leverage AI

4:13

or LLMs, uh, within their tooling. What

4:15

would be your advice to engineering leaders who

4:18

are thinking, Hey, I wanna start using

4:20

AI to extend the capacity of our product.

4:22

How should they kick off?

4:24

I think, let's start with ai. Just

4:26

ai. Someone wants to do AI

4:28

with their teams. Okay. That is a

4:30

huge pro, huge challenge because

4:33

if you look at all the leaders

4:35

in the industry, how do we

4:37

see if something is successful in the industry?

4:39

We have to get some ROI with our,

4:42

with all the endeavors, right? And

4:45

data science, AI, it's an experiment.

4:48

So when we start building it, it is always

4:51

not clear how this is going to show up.

4:54

For example, in AI models, we,

4:56

we try to build a model. We experimented

4:59

with historical data, and then we take

5:01

the model out in the market. And

5:03

then it starts getting all this new data from

5:05

our customers and our

5:07

model is giving answers

5:10

live in the run time. So

5:12

it's very difficult. It's an experiment that we are

5:14

running. Yeah. Leaders

5:18

should be very careful that they know

5:20

that what are the risks of running these

5:22

experiments. That's number one. Number

5:25

two, they should be very careful on

5:27

how they measure adoption

5:29

or how the results of these experiments

5:32

are being used by end users. Once

5:35

these two pillars are in place,

5:37

I think it's easier for leaders to measure

5:39

value and define ROI. Without

5:42

these two, AI, we can't do AI

5:44

or we can't do So

5:48

these are the two main, I would say, pillars

5:50

if you want to do AI. Now there are

5:52

other, uh, you know, organizational

5:55

challenges as well. For example,

5:58

democratization of data. Everybody

6:01

talks about it. But it's

6:03

a difficult job. Their data is

6:05

always in silo. There are different channels.

6:08

If you look at marketing, you get,

6:10

there is social media, there are emails,

6:13

there is YouTube or other

6:15

social networks. So, so many different

6:17

channels and you have to get the data together.

6:20

If you look at logistics, there

6:22

is carrier data in the ocean, carrier

6:24

data in the Um, and on

6:26

road, in, in air.

6:29

So it's just that if we

6:31

want to measure any process,

6:33

data is always in silo and

6:35

there is a huge, there are huge effort that

6:37

need, that is needed, um, on

6:39

the side of the person

6:41

who wants to do AI, to do a successful

6:44

AI. That's number one.

6:46

And we need, we have the architecture,

6:48

we have the solution, we know

6:50

that we need a data lake or we need

6:52

a source of data where all the data can

6:54

be consolidated. But it's, it's

6:57

a huge effort. It's a huge

6:59

foundational effort that always

7:01

doesn't show direct ROI. So

7:04

you have to convince as a leader in AI,

7:07

you have to convince your leaders that, hey,

7:09

I'm going to do this foundational effort. However,

7:12

you might not see a direct Right

7:15

now, but it's going to,

7:18

um, you know, executives talk about this

7:20

flywheel effect, but it's going to create

7:22

that flywheel effect at some point. So.

7:26

That's the discussion they need to drive a

7:28

Absolutely. I think it's really important for us to

7:30

understand both the potential and the risks

7:32

of ai. And, and I don't mean,

7:35

you know, the risks of a GI

7:37

and you know, this world

7:39

of like, oh, like crazy

7:41

cyber villain and creative ai like that. That's, that's

7:44

fun to think about Unci. Yeah, sure, we can talk about that.

7:46

But like, like more specifically to your business. There

7:49

is a challenge where if you don't put these foundations

7:51

in place, there's major risk to how

7:53

your business will present itself, whether that model will hallucinate.

7:55

And this is where it comes down to these foundational

7:58

data science concepts you're talking about, like, is

8:00

my data siloed? Do we have the

8:02

right training data? Is that training data

8:04

validated? Are there issues with

8:06

that data set that are going to cause long term

8:09

issues? And when I've talked

8:11

to other data science leaders like

8:13

yourself, that is one of the things people really hone

8:15

in on. So I'm glad you bring up this foundational piece

8:18

because A lot of leaders are getting

8:20

pushed by their board or pushed by their c-suite

8:22

of like, oh, we need to get AI in their product. But if,

8:24

if the data that you're feeding in to train

8:26

the model isn't, some data, isn't

8:29

data that is actually, uh, validated

8:31

and maybe peer reviewed or, or, uh,

8:33

checked on, like there are major risks that you put in play.

8:35

Yes. Yes, absolutely. You need

8:37

to know what your data can do. Without

8:39

that, you know, garbage in, garbage out.

8:42

You cannot save yourself even with an

8:44

LLM. So, yeah.

8:46

Very well said, and I think

8:48

it's challenging though for a lot of leaders

8:51

to get buy in on that foundational work

8:53

as you point out because there is an immediate ROI.

8:55

So, how should

8:57

engineering leaders start to, uh,

9:00

get that buy in about

9:02

ensuring they actually do all the steps needed to

9:04

be successful? Yes,

9:06

I think, we work in an agile

9:08

world, so we should have

9:11

a plan. We should have a plan with different

9:13

phases. And I always

9:15

say this to my team that each phase should

9:17

have a deliverable. It might not

9:19

be a specific product

9:22

or anything, but it should have a goal. It should

9:24

have a deliverable. I don't know

9:26

if I read about this Wabi

9:28

Sabi. It's a Japanese, uh,

9:31

Guiding Principle. It talks about continuous

9:34

development and that imperfection

9:36

is good. And I say this to my

9:39

team that while we say let's do something,

9:41

it is okay to be imperfect,

9:44

but let's have a goal. And when

9:46

we have a goal and when we have a plan

9:48

at each phase, then

9:50

big foundational work

9:52

might seem achievable. And that

9:54

is very important as Say

9:57

you are my boss, Conor, and I'm talking

9:59

to you, and I'm giving you a plan, and I'm saying

10:01

that, hey, phase 1, 2, I have

10:03

a goal, and I know I can go to phase

10:05

6, and at the end of, although

10:07

you cannot see an ROI at 1

10:10

and 2, but since I know I

10:12

can give you that ROI back, at

10:14

phase six, so that this is very important.

10:17

If I give you the plan that in that

10:19

case, you'll be more convinced than me

10:21

going to you and saying that, Hey, I don't really

10:23

know how to go to phase six, but we need to do

10:25

phase one.

10:26

So if I'm an engineering leader who hasn't

10:28

had deep experience with data science or ai,

10:31

and I'm thinking about how do I build this phased approach. Um,

10:34

what would be the general

10:36

steps you would advise, or is there a resource where

10:38

leaders can go in and say, Hey, let me look at,

10:40

I don't know, a template to start applying

10:42

to our specific use case?

10:45

Yes, I think there are many if you, if you look

10:47

at Google, like, how do you use, use gather

10:50

ROI for data science projects? But

10:52

I would say before we do that, it is very

10:55

important that in any organization, there is

10:57

a product counterpart with a data science

10:59

engineering manager. I believe

11:01

there should be other people championing

11:04

data science rather rather than

11:06

get So you in Yeah. To get buy-in rather than the data

11:08

scientists themselves. So it is critical

11:10

that you have a friend in the product organization

11:13

because they can look at the product holistically

11:16

from a top level view and they

11:18

can help you. Go ahead.

11:20

What would you say to people who are having trouble

11:22

finding that champion or picking the right champion?

11:26

Convince your boss, convince your boss

11:28

that you need a product partner. A single

11:30

person or a group of data scientists

11:32

can't always do everything. You need

11:34

to have a, someone else beyond that organization

11:37

who could champion for you. Sometimes

11:39

an executive can play that role too,

11:41

but with, with

11:43

having some time and common goals

11:46

and everything, I think it's critical that data

11:48

science organizations have a product partner.

11:50

And I'm sure this creates some translation

11:53

challenges across the company, as you're

11:55

trying to bring in these other stakeholders and get people

11:57

to buy in because you know you need the support,

11:59

but maybe the goals across

12:02

those different organizations can be different. Yes. Um,

12:04

how would you try to solve that

12:07

cross organizational translation challenge

12:09

to, to get these champions?

12:11

Well, I, I don't think the data scientists

12:13

can solve it by themselves. What, what

12:15

they can do is they, they

12:17

can say that, hey, I understand

12:19

it's late in your roadmap and I can

12:22

be ready on my part and I can make

12:24

it easier for you to understand

12:27

and access what I'm developing.

12:29

But I do think executives play a very

12:32

important role here because they need

12:34

to drive alignment across different

12:36

teams at the organization. Let's say

12:38

engineering team A and engineering team

12:40

B both wants to do data science, but

12:43

they don't have time because their roadmaps

12:45

are full of other projects. Data,

12:47

whatever the data scientists do, it won't

12:49

convince them, right? So the executives need

12:51

to prioritize that, hey, or, or a product

12:54

partner who can look at it and say, hey, this

12:56

data science feature, if we do it, this

12:58

will drive huge ROI than

13:01

a, than a small change or than

13:03

any other feature that we are taking out

13:05

this year. So we need, we need

13:07

to have executive alignment on roadmap

13:10

across teams and also. Some

13:12

other champions, but what the data scientists

13:15

or data science, uh, data science,

13:17

uh, organization leaders can do is

13:19

they can think of, okay, here

13:21

are some benefits of, um,

13:24

empowering data scientists and

13:26

data engineers so that they can write

13:28

code well. Um, this

13:30

is, I'm going off a tangent because,

13:32

you know, data scientists come from different backgrounds

13:35

and they are always not the best software engineers.

13:38

So they need support from data engineers and

13:40

they need to productionize their code, write production

13:42

level codes. So what the leader

13:44

in the data science organization do is make

13:47

sure that the organization

13:49

is empowered to Build

13:51

something that is very easily accessible

13:54

and can be taken by the

13:56

engineering team, and the engineering team doesn't

13:59

spend a lot of time building that or

14:02

understanding that.

14:03

It's interesting because you're talking a lot about these

14:06

change management concepts, frankly, of like,

14:08

you know, getting organizational alignment, building

14:10

up champions within the org, making sure you get

14:12

buy in, so you can showcase that ROI,

14:15

ensuring you have these phased rollouts and a

14:17

clear goal for each step of your process.

14:20

What if you're having trouble

14:22

getting that kind of buy in? Are there

14:24

ways that, you know,

14:26

data science or engineering teams can

14:29

leverage currently available

14:31

AI models or tooling to showcase the

14:33

ROI and then create that buy in?

14:36

Yes, there are also many

14:38

tools in the market that ROI,

14:42

but it's a little bit difficult. Once again, I

14:45

think, uh, It's an

14:47

experiment. That's how I see data science.

14:49

So it's not, depending on how much

14:51

customer, how much data your customer

14:53

have. You know, some customers

14:55

might be new. Some customers might not

14:58

be storing data very well. So

15:00

the experiment that I have run on

15:02

my end might not always

15:05

be great when I run it

15:07

with real time data with all my

15:09

customers. So having those risk

15:11

factors. Figuring those

15:13

out before release or having

15:15

a slow release so that you can talk to your

15:17

customers and figure out. Conor is

15:19

a great customer because we have five

15:22

years of data and our model is going to

15:24

give very good results. Whereas Mitra

15:26

might not be a good customer because she

15:28

has only six months of data. Figuring

15:31

those out and how, what is the fraction of your

15:33

customers will give good results. So the,

15:35

the risks, I think thinking those

15:38

ahead of time. Makes a huge

15:40

difference. I mean,

15:41

we've talked about this some on the show before about the importance

15:43

for leaders of understanding the risks of

15:45

even exciting opportunities. Yes, and I

15:48

think you bring up a good one, which is like, it's really easy

15:50

for us to over-exaggerate the

15:52

impact of AI on a particular customer

15:55

or on, you know, a feature or product

15:57

where maybe the realistic. Truth is that it's

15:59

going to take time for it to develop because you need that,

16:02

uh, data integrity to, to actually build

16:04

up, because you need more customer data. How

16:06

should you go about communicating with customers

16:08

about what you're able to do with AI

16:10

as you build your program?

16:12

I think building trust with customers is

16:14

key. Um, at

16:17

PagerDuty we have a process

16:19

called Early Access, where

16:22

our product is not fully built out, but

16:24

we are in the Early Access program, we have a prototype,

16:27

and we can Ask our customers to

16:29

use it and give us feedback. I think that

16:31

feedback is critical. They can tell

16:33

us that, hey, it's giving great results. They

16:35

can tell us it's giving very bad results. So

16:37

then we know and we can improve

16:40

and we can, so this early access

16:42

program is very useful.

16:44

How are you leveraging AI at PagerDuty?

16:47

We do a lot of AI. Um,

16:49

so We have five

16:52

features, and when I say features, these are

16:54

features which have different models in the, um,

16:56

in the backend. So we have five AIOps

16:59

features, and our AIOps, which used

17:01

to be an add on for our

17:03

full IR product, now is a separate BU.

17:06

So we have, um, A lot of

17:08

features in AIOps, noise reduction. I was

17:10

just talking to someone who mentioned that, uh,

17:13

uh, it's always a problem when you have

17:15

lots of, um, alerts.

17:18

And we are talking about security camera, like Google

17:20

camera. You keep on getting alerts and then

17:22

you are lost, right? So, the same thing

17:24

happens when people use PagerDuty. People

17:26

use PagerDuty when there is an incident and you are

17:28

getting alerts. And if you get a lot of

17:30

alerts, if you're inundated by alerts, you don't

17:32

know which one to go for. So we

17:34

have very good noise reduction

17:37

algorithms and we use AI

17:40

to, um, build those,

17:42

uh, noise reduction algorithm. That's, that's

17:44

a super smart use case. 'cause that kind of

17:46

cognitive load, it really makes us start to tune out.

17:48

I mean, like, I'm sure we've all been guilty of this.

17:51

Maybe with our email. Sometimes it's, ah,

17:53

so many, okay, I just gotta get through these things. Yes. And

17:56

it's so easy to miss something that might be important

17:58

if you're not really staying on it. Uh,

18:00

that's a great example of how you can leverage the power

18:02

of AI to assist for your customers.

18:05

Um, are there other ways that you see PagerDuty

18:08

leveraging AI in the future? Yes, we

18:10

have root cause. So these are, I'm talking

18:12

about, you know, non NLM AI

18:14

ops features. We have root cause,

18:17

we have probable origin. Uh,

18:20

what we do with these features is during

18:22

the incident, during the triage process, we try

18:24

to provide information to developers who are

18:26

looking We're trying to figure out what has

18:29

gone wrong, how can we figure

18:31

out, how can we resolve the incident faster.

18:33

So we have a suite of features

18:35

on that end. On the LLM side

18:37

of things, we have three new features

18:39

that are coming out. These are our first

18:42

Gen AI features.

18:44

We have a summarization use case.

18:47

I think this is a very good use case and

18:49

uh, one of the ways I always say,

18:51

once again going off a little tangent,

18:53

I always say that if you want to do LLM, find

18:55

a good use case. And I think this is

18:57

an awesome use case. So During

19:00

that incident, developers are trying

19:02

to solve, you know, a problem

19:04

and say that, okay, I'm resolving the incident. But

19:06

even during that phase, they have to update

19:09

their stakeholders or external,

19:11

uh, companies that, who are waiting for

19:14

information about the incident that is going on. That's

19:17

a very difficult job. Developers

19:20

who are always in the back end, they need to

19:22

write up an email and, you know, divide

19:24

their attention between solving a problem

19:27

and drafting up an email. So you'll give it

19:29

to the generative AI platform because

19:31

now they can do it for yourself. And

19:34

those conversations are already

19:36

there in Slack, in Zoom, in Microsoft

19:38

Teams. So why repeat it? Ask

19:40

your generative AI model to write it for yourself.

19:43

Smart.

19:43

So I think this is a very, very good use

19:45

case. and empowering to,

19:48

to the developers who are using PagerDuty.

19:51

And you mentioned this idea of ensuring

19:53

you find the right use cases or good use cases.

19:55

What's the approach that

19:57

you should think people should take to that?

19:59

What is the problem that you are solving for?

20:02

How would you provide relief to your

20:04

customers or

20:06

the end users? What would they find

20:08

useful? That's the key. And that's true

20:10

for LLMs too. You want to find

20:12

the use case, but is your use case

20:14

solving the problem that is being asked

20:17

by a lot of your customers?

20:19

And that's just good business advice, period, right?

20:21

Like, solve your customer problems. The

20:23

phrase, like, make your beer taste better has

20:26

been popularized recently. It's a great example. We have,

20:28

we say in PagerDuty,

20:30

champion the customers.

20:32

Great way to put it. I'd love to just get some more general

20:34

thoughts from you about your viewpoint

20:36

on AI in general,

20:38

where the industry's going, the explosion of success

20:41

here now that we have paralyzed GPU

20:43

models, we have years of them working, um,

20:46

obviously there's been an explosion in the public consciousness

20:48

of the ability to leverage AI,

20:50

and as you pointed out, like, this

20:53

all goes back to Like,

20:55

old papers, these, this goes back to

20:58

old sci fi novels, frankly, where we talked about

21:00

these ideas. Yes. And, and now we're

21:02

seeing them come into reality. Uh,

21:04

what are some of the things that you're excited about by

21:07

the current AI revolution that's happening?

21:10

I think I'm seeing very good use

21:12

cases. One use

21:14

case that I really loved. I am

21:17

big on Instagram. You know, I

21:19

was looking at the photo editing capabilities

21:22

and you can just take out a person

21:24

that you didn't like.

21:25

So if you've got an ex

21:27

boyfriend or girlfriend or something like

21:30

that, like, yeah.

21:31

No, think about it. I want my

21:33

picture before the Eiffel Tower.

21:36

And I don't want anyone else. And

21:38

I can do that now with AI. So

21:40

I love it. I love some

21:42

of the applications that are coming up. This

21:45

is a fun one, but there are very useful

21:47

ones in the, if I look around. Um,

21:50

recently when I was going to

21:52

the Chicago airport, they

21:54

did a, they did a facial

21:57

scanning. And they didn't actually

21:59

scan my boarding pass. Not LLM,

22:01

but still it's so cool. Where

22:03

I'm just walking and there is a machine

22:05

who is scanning my face.

22:07

I get some privacy concerns that I have to

22:09

admit, but it is very cool.

22:10

Yeah. Yeah, it's just so cool. Yeah.

22:12

Uh, well, thank you so much for taking the

22:14

time to chat with me today about this. It's been fascinating

22:16

to dive into your thoughts about AI. Do you

22:18

have any closing thoughts you'd like to share with the audience

22:21

about either how they should approach LLMs

22:23

or what all of this change means?

22:25

I would say data science leaders

22:27

fight for, fight

22:29

for space. Uh, you need to do more.

22:33

Uh, think of a good use case. Ask

22:35

your executives for a product partner. Try

22:38

to prove that this, the features you want to

22:40

develop, that the use case you are vouching

22:42

for, you want to build for, is going to solve

22:46

a customer problem. And that

22:48

is needed. Write up. I think

22:50

writing is very useful. And

22:52

give it away for people to consider,

22:55

take feedback. Be vocal. Yeah,

22:57

I would say that

22:58

Well said song. thank you so much for coming

23:00

on the show. It's been a distinct pleasure. If

23:03

you're someone who's listening to this conversation, uh,

23:05

consider checking out on YouTube. We're here in the midst

23:07

of an incredible lead dev conference, here in

23:09

Oakland, and I think it would be a ton of fun for

23:11

you to see us having this conversation live, uh,

23:14

on, on the YouTube channel. So that's,

23:16

uh, dev interrupted on YouTube, check it out. And,

23:18

uh, once again, thanks for coming on the show.

23:19

Thank you, Connor. Thanks for the invitation.