Nathalie Post  
In today's episode, I am joined by Jenn Gamble. Jenn is the Data Science Practice Lead at Very, and was previously data science director at Noodle AI and also holds a PhD in electrical engineering. In today's episode, we are going to be talking about building machine learning products. Now, that sounds like an incredibly broad topic. And I have to say, this episode is packed with insights, and therefore almost hard to summarise. So in the next 40 minutes or so, we are going to be diving into the data science development lifecycle, agile development practices, interdisciplinary team work, and also practices like ML Ops, test driven development and pair programming. And in a very human understandable language. So you don't need to be a programmer to follow along. Like I said, hard to summarise. So let's get into the episode. Enjoy.

Hi, Jen, and welcome to the human centred AI podcast. It is so great to be talking to you here today. I kid you not I have been looking forward to this for a very long time now. So for our listeners who may not know you yet, could you start by giving a little bit of an introduction about yourself and your background?

Jenn Gamble  
Sure. Yeah. Thank you so much for having me. Yeah, I know, we've, you know, chatted with each other for a while now. So I'm really happy to make it onto the podcast. I'm Jen Gamble. Like you said, I lead the data science practice at a company called Very. And so we are an IoT engineering firm. So what we do is partner with our clients, we build different types of products, many of them involve machine learning, or IoT components to them. And so often we're working in areas like smart manufacturing, or consumer IoT products, that type of thing. And what what brought me to this company in particular, is that I was, I had been working at a number of different kind of, you know, AI startups and enterprise AI companies in Silicon Valley, and had a lot of experience building machine learning products and working in kind of the traditional data science development lifecycle. And I became more and more interested in the type of best practices from software engineering and like, agile product development, that felt really relevant for machine learning development. But were not always the way that machine learning development was going in industry. And so I knew that Very, like, I met the director of engineering there, Jeff McGee, through he had some really nice online content about like machine learning, development, best practices, and this really kind of software engineering minded approach to it. And so, and Very takes their software development, like really seriously, and their agile development practices. And so bringing that together with kind of machine learning system development was what I was really, really interested in.

Nathalie Post  
Yeah, I find your career journey and the choices you made in those and you know, also kind of the, the reasoning behind those choices so interesting to hear. I mean, I cannot wait to ask you a lot of questions linked to ml development and, and kind of how you apply these agile development practices. So to start with maybe a very broad question. But let's say you want to build a machine learning system, where do you start?

Jenn Gamble  
Oh, my gosh, such a good question. Because I feel like so often, when we think of machine learning systems, and places that do this really, really well, we think of the big tech companies. And so, they often have very large teams, and lots of people to be able to not only, you know, do the kind of machine learning model development but you know, the, the pipelines and the DevOps and the ML ops and ml infra and, you know, there might even be whole teams, like Jjst working on advanced feature stores and having really high quality, high quality features available for the data scientists to use in their models. And they might have like a very well fleshed out model experimentation framework and ways to incorporate different models into, you know, staging environments and production environments for AB testing. And that's like, this whole machinery that kind of already exists in many of the kind of most advanced tech companies, that then individual, you know, contributors can work within. But I think that, totally opposite from that, is the context that a lot of companies find themselves in where, you know, they may be in tech, they may not be in tech, but they're really just now taking, you know, data science and machine learning, and like data driven decision making seriously as like a core component of their business. And so they've, you know, they've hired some data scientists, they have some initiatives, maybe they've done some, you know, proof of concepts or something like that. But they definitely don't have this huge machinery that is like a well oiled machine. And so they're trying to start, you know, kind of from scratch. And so this is, this is the area that I, that I find some of the most exciting stuff happening right now is like, if you are just, you know, a team of one or two or three data scientists, like, Who else do you need to work with? And what do you need to be able to do to actually stand up a machine learning system, in like a production setting, the people are really using those outputs on an ongoing basis?

Nathalie Post  
So who do they need to work with?

Jenn Gamble  
Yeah, right, perfect. There might exist some type of data scientist unicorns that can do the majority of what's required by themselves. But they're, they're definitely not a typical data scientist that you would find in industry right now. So I think one of the most important partnerships is between data scientists and software engineers. Hopefully, over time, we'll have more and more data scientists that have stronger and stronger, kind of like software development practices. But right now, I think that pairing together data scientists and software engineers on a project during the development phase can be extremely useful. Because the data scientist is really thinking about, you know, what's required, from the data perspective, from the modelling perspective, like, what do we, what do we need to be doing to the incoming raw data and various pipelines and all of this, but they might not be asking the right questions about, okay, what's the right abstraction to use? Like? What are the right kind of modules that I need to be able to compose together to make this pipeline? What are my deployment processes? How do I have this automated? What types of tests do I need to add? Like all of those questions, they might not be as used to asking. And so having a data scientist and a data engineer worked together really tightly on the development of these machine learning systems, we found can be can be really valuable. Another key person I would say in this is like the designer. Sometimes these people are designers, sometimes they're  front end developers who have a really strong focus on like the UI and UX aspect of it. Or sometimes these are, like product managers, who are, you know, in really tight communication with the customer, the end user, like, what we're actually providing as value in the product and how machine learning is like a key enabler of that. Definitely a lot of the a lot of the core principles from user centred design are extremely valuable for data scientists. And again, it's an area that not all data scientists are very fluent in. But having a really clear understanding of, you know, who my end users are, like, what information are we providing to them? What decisions are they making or actions are they taking off of this information? Like, what are the end user flows that we're actually trying to support? Sometimes, like people, in the back end doing technical work, think that they don't need to have a really deep understanding at that level of the product. But I would definitely advocate that they they can and should have a deep understanding of that. But they're also people who have kind of a stronger background in UI and UX and design can be kind of partners with data scientists, during the development to help them make sure that they're thinking about the right things as they're working.

Nathalie Post  
Yeah, I mean, yeah, so I think it is really interesting, because I know you mentioned it briefly before, but like, how do all these different practices then come together? And like perhaps even to start with? In what way? Is the data science development lifecycle different from a more traditional Agile software development lifecycle?

Jenn Gamble  
Yeah, I'll take maybe like, two components of that question I'll answer in two separate parts. So one part that's more about more about the team and the composition of the team and the way that they come together. I would say that this is one of the trickiest parts about building machine learning products is the type of interdisciplinary teamwork required, and the way that we need to be kind of cutting across so many different stacks. I mean, even at my company Very, like we're also incorporating IoT into the mix. And so we have like some of the areas that I already mentioned with, you know, machine learning / data science, and you know, software development, UI UX. Sometimes you need some explicit skills in terms of like cloud operations, like Big Data pipelines in the cloud, and a lot of DevOps, stuff there. Sometimes you need mobile app skills. And then also on the IoT side, you often need like hardware development, and like firmware skills, too. So you might need to have, at a minimum, like six or seven different skill sets, kind of like coming together. And each of those is still a pretty large bucket like I'm bucketing together, like all of back end development into like one big, one big like persona of like software engineer, right. And clearly, there's a tonne of tonne of nuance within that same thing as like data scientists, there's a tonne of different, you know, areas of specialisation and sub skills. But even talking at the level, like the high level of disciplines, trying to get these people to work together really effectively can be one of the biggest challenges because people with different backgrounds always, you know, they have different assumptions, they use different terminology to mean the same things, that it's almost like they're talking different languages sometimes. And so I find some of the things that help the most in terms of having these interdisciplinary, technical teams work together, like really effectively. One thing that's super important is having a strong tech lead. So sometimes this person is like the, like a technical programme manager or technical product manager. And they're like, focused on the product side of things. But in many cases, those people maybe have other priorities. And they're, they're focused on, you know, managing stakeholder relationships, and, you know, roadmaps and things like that. And so the person who I would call like, the tech lead, is really focused on like, the overall architecture of the system. What are all of the different components that need to fit together? Like, how are the interfaces being specified? What are the feedback mechanisms, all of that? And how do we kind of pair the understanding of the kind of product requirements with this kind of like technical architecture, view of the product. And then you have all of the individual developers focusing kind of on their own, they have each their own area of focus, which is usually like, you know, one or more components of the system. And they're really focusing at the level of implementation detail. And so one key thing there, I think, is that making sure that people have enough visibility, like not only into their own component, but the adjacent components, and the overall technical architecture, as well as the kind of end user like product requirements. So there's kind of like three levels of granularity here. There's like the product definition. And usually, there's like a product manager in charge of that. The second level of granularity is this kind of technical architecture level. And the tech lead is the person who's really in charge of that. And then the third level of granularity is like this kind of implementation detail level. And each kind of developer who is in charge of their own component is in charge of making the decisions there. But it's important to make sure that those people have enough visibility into the components above them and enough context, that they're not just receiving requirements and taking them as set in stone, that they can actually kind of give and take and have this these feedback loops where, you know, for example, a product manager says, Hey, we have requirement x, that by the time it gets down to an individual developer who's deciding on implementation decisions, they understand enough about what they're trying to achieve, that they can actually push back and say, Okay, I know you asked for x. But if I actually give you something that's very similar but slightly different, then this is why I think it's still going to meet your users needs. But it's actually going to save us like 80% of the implementation time required, or better word has all of these other technical trade off benefits. And so making sure that people have that, that type of understanding at all three levels, even if they have a specific thing that they're focusing on. So super important role. And I would also say, like, the collaboration amongst the team is extremely important. So often, people tend to get siloed, especially if they're, if the other parts or other components of the system feel like it's not in your wheelhouse or like, like, oh, designers are doing their thing, like I'm doing my thing, like, we'll just define, like, what are interfaces, and then we can each, we can each go off and do our work and come back together when our pieces are ready to try and put together. But I would definitely encourage people to be pairing more often across disciplines actually, like sitting and working together, and like looking at the work that each other is doing. And to be coming together and making the interface specifications like more of a more of a conversation, like, especially in early days, and as if he was trying to stand up like a new product, that you're not like working on like a big existing system, then being able to have that same type of like, give and take technical trade off discussion, you can end up building a full system that's just a lot more robust and well architected. As opposed to when each person is kind of working in a silo and just taking taking as set in stone, the kind of interface specifications and requirements that have been given to them. Yeah,

Nathalie Post  
Yeah, yes. So who typically facilitates these conversations and, you know, like, basically these pairings collaborations, what type of person or role is doing that, or should be doing that,

Jenn Gamble  
in my experience, is usually the tech lead. And so often, I find like a product manager will be mainly in charge of making sure that everyone is clear on and aligned with respect to the features that are trying to be developed, and the kind of timelines and roadmap and all of that, and then how that translates down into, like, you know, more specific individual level work. But then the tech lead is a person who's really kind of setting the tone for the development practices on the team. And so definitely, like collaboration, and pairing is one type of development practice that is, is really set by the tech lead. But also things about, you know, what types of like agile and DevOps practices are followed, and like the CI CD workflows, and like, how much how clear of documentation is required with your code, and how much testing is required. And you know, how much automation is required all of those things. I think that that role is really, really key to the, to the success of a project. And then also, it can be kind of a company culture thing to where if someone is working independently, how easy they feel it is to, like, reach out to others and ask for help, especially if they're working on something that is, like, new or unfamiliar to them. Or if there's like a different part of this system that they need to interface with, that they don't feel that they understand as deeply as they would like to how comfortable does that person feel like reaching across or reaching across the kind of silo lines or trying to actually blend the silo lines to, to have that that communication happen?

Nathalie Post  
Hmm. Yeah, so maybe just a silly, broad question. But what are your thoughts on agile in general? Like, like, How do you feel? I mean, even about the word, but maybe also just in the context of how it translates

Jenn Gamble  
Maybe I shouldn't comment on the word itself. Because, I mean, it's like, it's even worse than the word data scientist in that it means like 80,000 different things depending on who you ask, like they have their own mental model of like, what they think what they think a data scientist is or like, what they think agile development means. And, you know, the word itself is kind of meaningless at this point as like a buzzword. But I do think there is actually a lot of value in a lot of the underlying principles or philosophies that I think agile development is is meant to be, at least my interpretation of agile development is meant to be conveying, like, take, take test driven development as an example, right? Like, the basic principle is just before you actually go and implement something, first say, Okay, how can I tell if it's going to be working? What would I need to check to say, yes, this code is doing what I expected to be doing. And then to actually, like, write that test to perform that check, before you even go and implement the code to do the thing, right. And so following those types of principles, I think can be extremely valuable. Like test driven development, for example, you can you can apply those same types of principles at multiple levels of granularity throughout your system. Like in traditional software development, you have you know, your, your unit tests and your integration tests. And that's kind of the multiple levels of granularity that you're talking about it. And obviously, there's more like regression tests, etc. But I'm in data science, like trying to trying to follow the same philosophy, you start branching into, like more different areas, like what does it mean, to know that my data pipeline is acting in the way that I'm expecting it to be acting? Like, you start to think about data validation, and like, monitoring and being able to, Okay, how can I tell if my model is performing in the way that I expect it to be performing models or non deterministic, it's not like a unit test where you can just say, for this input, I expect that output, you need to be a little bit more creative, and like, what are we actually checking for? But, but having that mindset of defining what it means to be working, before you actually build something, I think can be kind of carried through throughout one of the other key philosophies from agile development, like this kind of CICD type of philosophy of, you know, continuous integration, continuous deployment or continuous delivery. Again, trying to apply that like, all the way through your data science development stack. This is where you get into a lot of the kind of ML Ops content, it's like another buzzword from these days. But there's just a lot of nuance to it. And and, like I so I would all of this to say that I totally agree with the foundational agile philosophies. And that it's not always just a vanilla application of them, when you're considering the kind of data and ML intensive systems that there's some kind of extra extra layers of nuance that you need to include in many cases.

Nathalie Post  
So you mentioned very briefly, like the the test driven development part and ml ops. And I'm kind of curious, like, what other types of practices do you apply and find helpful in building ml products? And actually, also, can you even explain what ml ops is, for our listeners who might not be familiar with this concept?

Jenn Gamble  
I would, I would define ML ops as just the extension of those principles from DevOps into building machine learning systems. And so like I mentioned, test driven development, I mentioned, kind of automated deployment processes and fast iteration cycles, I would also include with that, like version control, reproducibility, notions of like being able to, like unwind changes, or like go back to previous versions of the system. And I would also probably include, like logging and monitoring in there as like a foundational principle of being able to tell if something is going wrong with the system that's deployed. And so in the ML context, the biggest difference is that your system doesn't just depend on the code. It depends on the code and on the data. And so a trained ML model is this kind of artefact that is the output of a training pipeline, where the inputs are both the code and the training data that are used to produce it. So there's like, there's more kind of orchestration, layers and loops that are required in ML systems? Because it's not only Okay, what is my code? How do I get it into, you know, the different environments that it needs to be deployed to, but there's also this extra, okay, what is my training process, what data is used for that training? And then once I perform the training and have this model artefact, what is the deployment process for putting that model artefact into kind of my rest of my working system, and so you just need to kind of layer on these principles like testing and version control and automated deployment and logging and monitoring and like fast iteration loops throughout all of that into this whole, like not only code world, but code plus data plus models, world.

Nathalie Post  
Yeah. And then like, could you explain actually how this would apply to smaller teams, because, you know, we talked about a bunch of different practices. And it feels like a lot. And it also feels like you'd need at least a handful of people, or maybe 10 people to begin with. But I also know that this is often not the reality. So in your experience, how does this translate to smaller teams?

Jenn Gamble  
Yeah, right. So I think you do need a certain level of ambition in your, in your data scientists probably, like definitely, the style of data science that we practice at Very, is, is much more full stack than maybe you would see in many places, much more, maybe even like extended full stack in some cases. So like, we do have our data scientists, you know, setting up their own, you know,  CICD, and, you know, infrastructure as code for the different cloud resources that they're using, and handling the deployment of all of that stuff, as well as like setting up their own data pipelines, and, you know, model training and evaluation and kind of all of the pieces. But another option, also, as I mentioned, is this kind of pairing between data scientists and software engineers. And, you know, our data scientists and software engineers also do work really closely together. So one thing that we definitely find helpful is getting an end to end version of your system up and running as early as possible. And so it's kind of opposite from the way that data science has traditionally been done, where often you'll have some data, scientists spend a really long time in kind of like an exploration phase, where they're figuring out really like what's even possible with this data, there'll be exploring it, figuring out what type of transformations they need to do, what type of modelling approach they should use, like, they spend a lot of time on feature engineering, and like hyper parameter tuning. And then at the end, they say, Okay, I'm happy with this model, let's find a way to productionize it, let's find how, how we need to set up our like proper, you know, production pipelines for training and inference, and like, get the whole system running kind of for real. And so I advocate almost the opposite approach where, where you start out very, very early, trying to actually get the whole thing running end to end. So for your model, this includes like the training, evaluation, validation, serving, maybe even monitoring, and your data pipelines, like there's whatever extraction, analysis preparation, those those eight steps are described, there's a really nice document that Google has on ML Ops, I would definitely recommend it to anyone who's interested in this field, they also have a nice rules of ML document that's a few years older, but both of those are have some really solid material in there. But anyways, saying that getting that actually running end to end with like, as simple things as possible, at each point in the system, can be extremely valuable. So you're like, I know that this is not going to be useful in the end, but I'm just going to throw up into whatever like my staging environment, for example, I'm just going to take, you know, some bit of extract out some small set of the data, I'm going to run it through a quote unquote, pipeline, that is just maybe it's not even doing anything, it just has some like identity functions, or it's just doing some like very simple aggregations. I'm gonna run it through a model that's just like, you know, a linear model, or like the most basic thing you can think of, and I'm going to have some outputs, I'm going to evaluate those outputs, going to choose some metrics, whatever those mechanisms are, and actually having the whole thing running end to end. It forces you to think in a much more modular way, because you know that every single piece of it is going to need to be swapped out for an improved version of it. It forces you to pay attention to interfaces, pay attention to those kind of development and deployment loops, where you're actually like, because you know, that you need to be going through these loops so many times and you need to be updating, and redeploying and changing and adjusting and evaluating constantly. That the earlier you can like get those loops going and start exercising all those different parts of the system then over time, it just becomes easier and easier. And the system that you end up building is so much more robust and easy to update and improve and continually update improve, as opposed to the opposite tact, which is often taken of like waiting until you kind of have what you think is a good version and then putting it up. And then after it's up figuring out like, oh, how would I update this if I needed to?

Nathalie Post  
Yeah, I mean, it almost sounds like a bit of the opposite of a waterfall approach where, you know, you're trying to get like every part, right? And it doesn't often happen.

Jenn Gamble  
Because the thing with waterfall and like the reason why people who I advocate for agile, say that you shouldn't use waterfall, it's not that you don't want to plan right, you definitely want to do some upfront planning, but is that if you've built a lot of systems before you know that you're going to be uncovering new information as you go, like your upfront plans are not going to be perfect, I can guarantee you, right? And so don't build into your development processes, this knowledge that like, what you think is the correct thing to do is going to need to change over the course of development, then it's just it's a lot harder to make those updates if you weren't already planning your work in that way.

Nathalie Post  
Yeah, absolutely. And I mean, I also really think it feeds into this whole mindset about these things, you know, of like, not falling in love with the solution and being very fixed and attached to what you've build, but instead have this mentality of like, No, no, this is for now. And we're gonna do it again. And again, and again, rather than we're gonna make this one thing, no such thing.

Jenn Gamble  
I and this also comes from, like a product development mindset, too, I think because people in product, they generally have that mindset very well, where they know that the first things that they kind of get out in front of their users are not going to be what the users actually want. And that the only way to find out what the users really want is by actually putting something in front of them. And iterating more and more, this is like the whole kind of like MVP style development approach. Right? And so like, how to take that into our machine learning development? Definitely a key part of it.

Nathalie Post  
Yeah, yeah. Cause in that sense, like, how do you integrate those other core competencies, so to say, in that entire process, while you're building a machine learning model, so you know, in your role at Very, what do you know designers typically work on? What does the let's say business person? Do at that time? Like, what does their role look like in the projects that you work on?

Jenn Gamble  
Yeah, so one thing that we, we aim for as much as possible, is working in like, very skinny, but very tall, like vertical slices. And so like one, one feature that we're trying to develop, we'll cut through like many, many different components of the system, there might be some data science parts, back end parts, front end parts, like design parts. And so and, and working in this way, has a few benefits. One of them is that this is a mechanism by which you kind of force the whole team to communicate with each other much more regularly. And effectively, if each of them is working on like, you know, a much bigger and more horizontal part that's contained only within their own component, then they don't really have a lot of reason to need to be talking back and forth with each other as much. Whereas if, like, they're all working together on like, okay, we know that we need to update the user flow for this one kind of analytics part of the dashboard. And we're going to be changing some stuff, not only about the backend and the way it's processing, but what the data science pipeline is outputting into the back end, and the way that the front end is ingesting that and surfacing it to the user. And having like that entire set of people working together for like a whole sprint, for example, on this component, that that can be a really effective way and, and getting, like getting what you have all the way into production kind of as early as possible. The second thing I would say is, you do always want to be very clear on like, what the, what the feature is, like the end user feature is that you're trying to improve or create with the individual work that you're doing. Often people that are kind of like further into the back end or like more layers under the hood, we we have a harder time saying exactly like, Oh, this is what is going to be better because of this work that I'm doing here. And so how having that really clear clarity like often for data scientists, I would say, even if you could have work with a designer and have an idea of like, what, what the actual, like wireframes would be like where in the front end? Is this information going to be getting surfaced? Like, what is the user going to be seeing? How are they going to be receiving this information and like taking action off of it can really help them, make sure that they're even framing the problem correctly, analytically. So and it also makes sure that they're only doing the work that is actually required, that can be one of the biggest, biggest challenges in a system is not actually building something that works. But like getting there as efficiently as possible. Often people do like a lot more than what is actually necessary to get to the end product that they're trying to build.

Nathalie Post  
Yeah, I mean, I think those are some great closing words, cuz we're getting towards the end of our time here. But to wrap things up, I always like to ask the question, if if someone listening would only take away one thing from this conversation, then what would that the?

Jenn Gamble  
Oh my gosh, DevOps is for everyone.

Nathalie Post  
This is great. Well, we'll keep it at that. So then, finally, finally, as the last question, if people want to learn more about you, or about fairy Then where should they go?

Jenn Gamble  
Yeah, sure, our website is verypossible.com. And we have a really nice blog series. And so if you search for my name, Jenn Gamble in there or just any of the ML tags or if you're interested in other stuff related to our IoT development, there's a lot of a lot of content in there that you should be able to find.

Nathalie Post  
Great. Thank you so much, Jenn. This was amazing. Thank you.

Jenn Gamble  
Thank you so much for having me.

‍