There’s a lot at stake in the decisions that social workers have to make when they care for people – and Dr. Besa Bauta keeps this in mind when her teams are designing the data products that care providers use in the field.
As Chief Data Officer at MercyFirst, a New York-based social service nonprofit, Besa explains how her teams use design and design thinking to create useful decision support applications that lead to improved clinician-client interactions, health and well-being outcomes, and better decision making.
In addition to her work at MercyFirst, Besa currently serves as an adjunct assistant professor at New York University’s Silver School of Social Work where she teaches public health, social science theories and mental/behavioral health. On today’s episode, Besa and I talked about how MercyFirst’s focus on user experience improves its delivery of care and the challenges Besa and her team have encountered in driving adoption of new technology.
In total, we covered:
We’re not a technology company, …so, for us, it’s about finding the right partners that understand our use cases and who are also willing to work alongside us to actually develop something that our end-users – our physicians, for example – are able to use in their interaction with a patient. – Besa
No one wants to have a different type of application every other week, month, or year. We want to have a solution that grows with the organization. – Besa on the importance of creating a product that is sustainable over time
If we think about data as largely about providing decision support or decision intelligence, how do you measure that it’s designed to do a good job? What’s the KPI for choosing good KPIs? – Brian
Earlier on, engaging with the key stakeholders is really important. You’re going to have important gatekeepers, who are going to say, ‘No, no, no,’ – the naysayers. I start with the naysayers first – the harder nuts to crack – and say, ‘How can this improve your process or your service?’ If I could win them over, the rest is cake. Well, almost. Not all the time. – Besa
Failure is how some orgs learn about just how much design matters. At some point, they realize that data science, engineering, and technical work doesn’t count if no human will use that app, model, product, or dashboard when it rolls out. -Brian
Besa: It was a dud. [laugh].
Brian: -yeah, if it doesn’t get used, it doesn’t matter
What my team has done is create workgroups with our vendors and others to sort of shift developmental timelines […] and change what needs to go into development and production first-and then ensure there’s a tiered approach to meet [everyone’s] needs because we work as a collective. It’s not just one healthcare organization: there are many health and social service organizations on the same boat. – Besa
It’s really important to think about the human in the middle of this entire process. Sometimes products get developed without really thinking, ‘is this going to improve the way I do things? Is it going to improve anything?’ … The more personalized a product is,the better it is and the greater the adoption. – Besa
Brian: Welcome back to Experiencing Data. This is Brian T. O’Neill. Today I have Besa Bauta on the line. We’re excited to have her on the show to talk to you about her work at MercyFirst as the chief data officer.
We’ve been trying to get this on the books for a while, and then this global pandemic hit, [laugh] which changed your job a bit and I know you had picked up some data governance work, and sounds like two jobs at least, if not three. So, I really appreciate you making the time. Welcome.
Besa: Thank you.
Brian: And first question I wanted to ask you because we were talking on LinkedIn about how your team goes about building analytics solutions, and data products, and delivering insights to MercyFirst, and whenever I see the D-the CDO-I’m always like digital, data, or both? And what’s that relationship? And how do you see it? Because you can’t really just do data without doing digital. Is that a meaningful conversation to have? How do you see your role as delivering these products, and services, and insights to the business in terms of digital?
Besa: Oh, I agree. I think digital and data, kind of, go hand in hand. I think data-you know, you heard this statement as far as the oil. Google had a presentation where they’re like, data is the next oil in the marketplace. I would like to think of digital as being something greater.
It’s not just the information or the collection of information. So, data is basically information. I think we’ve used different nomenclature over time. And currently, it has become the parlance as the new buzzword out there. But collecting information, using information to either improve businesses or practices has been around.
What’s new in the field is the whole process of digitization, basically leveraging a lot of the information technologies to get greater insights and actually move information at a faster pace. I think data is the basic element of digitization, or digital aspect is what you bring value to the whole enterprise. So, I see those as two separate terms, but they are interrelated. One is the essential component, and then the other one is sort of more enterprise-wide element that’s important in improving processes and systems.
Brian: One thing, so I kind of break this world-in my own consulting and training in the stuff that I do-and maybe this divide doesn’t make sense, but you have the digital-native companies, the tech companies where they’re selling software, or they’re entirely online; and then you have the traditional companies doing other types of work that have internal data, quote, “Teams,” or digital teams. And the way they make software tends to be different. I’m totally generalizing: it typically feels behind slower, not as effective as the way tech companies, which have had 20 solid years now of failing and learning better ways to do this. How do you go about making sure that the solutions, when they’re presented to your internal-or maybe you can tell us a little bit about who the users of your data products are at MercyFirst, how do you make sure that it’s actually working, that it’s providing value, that decisions are being made with the data, all these things that this last mile is-as I often call it-what do you do to make sure that people will use this stuff because the failure rates are really high in our industry.
Besa: So, I mean, I like to look at it slightly differently. I think you’re correct. You have technology companies; that is their bread and butter to sort of develop systems and processes to, you know, whatever, improve either financial functions, improve insurance, or improve whatever other sector. So, they are the developers and we’re the consumers in that process. I mean, I looked at it that way.
As a social service or a health organization, we’re talking technically not digital-natives in a lot of ways. Obviously, for insurance-related purposes, patient outcomes, we have to collect a lot of that data. But our focus is slightly different. I mean, our focus is slightly on improving health and well being. So, the systems have to be in place as part of that pathway for health and well being, obviously, for reimbursement and payment schemes.
And I don’t think it’s a fair comparison: us with technology companies. I mean, definitely we’re failing forward, but it’s their bread and butter to develop these technologies, and it’s much more integrated into the way they work because they are developing them, they’re utilizing them, and iteratively making them much better over time. However, and they will not have a customer. We’re their customers in a lot of ways, and their solutions have to meet our needs. So, let it be an insurance company where they’re improving processes for their partners, or for us in our patient profile, in a way, whatever solutions that we have has to meet our needs in order to improve outcomes.
So, for us is that we take multiple different solutions to meet our needs, there is not one unified solution. So, if you have a technology companies, they do have a lot of product and portfolios, but in a way, it’s more centralized. For us, what we’re doing, especially in the healthcare sector, is sourcing different vendors, sourcing different application to meet our needs, but they’re almost, like, retrofitted. They’re not homegrown. Actually, a lot of organizations that are healthcare and social server organizations are developing their homegrown because they don’t see anything else in the marketplace that actually meets their unique environments or needs.
And we’re learning. Obviously, we’re not technology companies, number one. So, for us is finding those right partners that understand our use cases, number one, but also are willing to work alongside us to actually develop something that our end-users, for example, our physicians, are able to use in their interaction with a patient.
Brian: Mm-hm. So, it sounds like you’re doing a fair amount of getting your partners that have off the shelf, and then doing some type of level of customization around your particular-
Besa: Correct.
Brian: -needs and all that. I’m curious, what’s the process of helping them-like, at some point, someone has to, typically in a tech company it’d be a product manager, but someone’s going to own a backlog of, quote, “Here are the changes or the modifications we need to make to satisfy Besa’s needs that your organization.” Somehow that decision needs to get made about the order of those things, which things they’re going to say yes to, which things are going to say no to. What kinds of activities do you do to go figure out which hungry mouth will get served within your organization the most, who’s going to have to deal-“Sorry, you get off the shelf Tableau, or whatever. We’re not going to give you any custom dashboards, you just get Plain Jane, start from scratch. But over here in operations, you guys get gold level service.” How do you figure out who needs what and designing those solutions and then getting them into the funnel to do the engineering and the data work?
Besa: So, I mean, for us, let’s say even when we’re selecting our electronic health vendors or other partners is actually working with them to understand our needs. But they have a business; usually, technology companies have a product they don’t want to sell and don’t want to customize that product so much for a sector, otherwise that then you lose the ability to actually sell it to the next partner. And then you don’t want to sink in a lot of development time, especially for, like, one user. You want to have an application that can go across users. So, for us, it’s sort of working with our vendors and other technology partners to understand our needs, number one, but allow us to have a transfer of knowledge.
So, if you’re going to give us a baseline package, provide that wraparound support for us to do that customization or that engineering support that meets our needs. That way we’re purchasing a product, but you’re providing the right support for us to continue and consume this product so it’s sustainable over time. No one wants to have a different type of application every other week, every other month, or every year. We want to have a solution that grows with the organization. So, that’s step number one. I think step number two, that’s a really important question: who gets the best application? Usually, who pays for it? [laugh]-
Brian: Yeah.
Besa: -oftentimes. Or who screams the loudest? It really depends. For us, we have a different type of approach. We have different divisions, and each of those divisions have different needs.
So, for us, there’s only so many projects we could handle at one time. And we would have to think about which project brings the greatest return on investment, or what has the shortest from working as far as taking it from concept to development. And then what we found is that sometimes the longer projects that take much more time, and energy, and things end up fizzling out, so what we have done is actually picked a few projects that have small wins in order to demonstrate organizationally that, yes, this is the right path, that look at the return on investment here. If we could duplicate this across the spectrum with higher needs processes, then you would have this much return on that investment. But it’s almost like a colleague of mine is like, “How do you eat an elephant?” And I’m like, “One piece at a time.”
Same thing for us is that you tackle one project at a time, and ensure with a smaller project that you get those types of outcomes. The bigger the projects are, there’s too many wheels moving that you really don’t know if you’re having the right impact, number one, but also the project management takes a greater role. And sometimes organizations do not have the capacity for larger project management.
Brian: Yeah, I often talk these days about product management and not just project, especially if you are developing custom solutions or there’s a long term vision for how off the shelf needs to be customized. You’re effectively talking about the role of product just as much as you are the day-to-day scrums or whatever methodology you’re-
Besa: Yeah.
Brian: -using to build out stuff. How do you measure the success of these tools? If we think about data large-and maybe you don’t think of it this way-as largely about providing decision support or decision intelligence, how do you measure that it’s doing a good job? What’s the KPI for the KPIs, like? [laugh].
Besa: I think we talk a lot about KPIs and, sort of like, short wins, but really to see if they’re really having outcomes, you need a really good study. And I haven’t seen that many excellent studies out there, like randomized control trials that actually takes to task, what we are implementing and does it have an impact downstream? For us, I think it’s just understanding, does it improve the processes for our physicians? So, a physician is meeting with a patient, having them navigate the electronic health record where they’re spending 45 minutes with a patient, instead of spending 15 minutes, I think that’s a small win. So, for our end users, I think one way to evaluate it is, are you bringing valuable support or technologies that improve the way they’re delivering service? I think it’s key.
So, if our providers are seeing a value add in their workflows, then I think that’s a really good KPI in and above patient outcomes. So, if we do a better job of supporting our staff, and supporting our frontline providers to provide the service, and I’m assuming de facto, that, through that better interaction, that the patients and clients have a better experience, and they’re not fidgeting with moving in from screen to screen to screen, and they’re actually looking at the client, that’s a win. And it’s also a win for the client. I understand there’s, for us, we’re in a world where everything gets measured, everything gets tracked, but not everything is important in the end. And sometimes that human interaction in that process is just as important.
So, I see the digital tools and the data as an important element, but they’re not the whole picture in that process of providing care for us. And in order to improve decision making, I think that individual interaction needs to be improved between our clinicians as well as our patients. And if that gets improved, then I think our outcomes and decision-making gets improved as well. And also sourcing the right information for clinicians so they understand what they need to target at that point of time.
Brian: Got it. So, you actually mentioned a couple things that I would consider user experience metrics that could be tracked. So, reduction in the time spent, either using software in a patient-doctor setting, or just a shorter time overall, the entire appointment is shorter, or it’s lower tool time, right, less screens, less time. They literally use the iPad for less time, but at the end of the day, they push the button so we know the record was updated, but it’s 10 times faster than it used to be. Are those things you measure either quantitatively or even qualitatively through some kind of design exercise or usability evaluation? Or who does that work? And how do you-
Besa: My team does that work. [laugh].
Brian: Oh, your team does that work. Okay.
Besa: Yes, my team does that work. So, user experience, user interaction is really important, as you’re going to be implementing a lot of these technologies, that you want to ensure that they have the right uptake, as well as sustainability long term. That’s important. And recently, I saw a really interesting application where somebody has developed an AI application where the physician just talks and the AI application basically is able to determine if it’s a patient, or a physician, or a nurse in that interaction pathway and actually enter that automatically into the electronic health record. So, that would save a lot of time.
So, the data capture aspect of it I think it’s fascinating. It’s a necessary evil; we have to deal with it, but is it possible to sort of leverage some of these latest technologies to take that burden off of our clinicians, or off of the providers so they could have that natural interaction with the data capture? Obviously, that creates a lot of ethical issues [laugh] but it’s definitely an interesting concept to think about.
Brian: Yeah, yeah. I think when we first connected, we were talking about design, and you had mentioned Apple, and it sounded like your team had gone through some training. And it sounded like there was, you said, it was difficult getting buy-ins from some of your customers, and I was curious if you could kind of unpack what you were talking about if you remember that context from our original chat.
Besa: It is always hard to get buy-in, especially with something new. Like, we’re interested in sort of bringing in biosensors and other sensors-especially during COVID-remote temperature sensors, and the conversation that, especially now, where will this data go? Yes, it’s definitely an easy, sort of, collecting temperature for anybody that comes in or collecting all of this information, but it’s definitely a new technology. So, there has to be a process of adaptation regarding, how do you use this technology? For what purpose, number one, and is it really going to improve my health and well being, or my life, or whatever, work processes?
And people always have fears? I mean, with anything new, there is always the fear, what does this really mean for me? What does it mean for my safety, for my well being? And will it really improve the outcomes the way that we have stated? And those are always things that I think about all the time as well. If I’m getting an Apple Watch, or if I’m getting the latest and greatest Alexa in my home, you worry about those things. And sometimes they’re rational fears, I understand that.
Brian: It sounds like in this context then, when you talked about users, you were talking about, actually, patients, so patient-
Besa: Yes.
Brian: -facing technology, as opposed to internal solutions used by your staff, or-
Besa: Both. [laugh].
Brian: -healthcare providers, et cetera. Oh, so it’s difficult on the internal side, too?
Besa: I think it is difficult on the internal side as well, and the internal side is sort of adopting new technologies for workflow processes. So, we have a recent project that we partner with an organization to bring robotic process automation, in a way taking what some of our staff do as far as data entry, and just getting rid of that human-in-the-middle process, and just automating data entry from one system to another. And obviously, for a lot of our staff, that’s also fear. “You’re going to have a robot take my job.” It’s like, the robot is not taking your job. The robot is actually facilitating that process.
So, in a way, it’s taking something that’s redundant and allowing you to actually concentrate that time on something much more meaningful and useful, rather than a task that can be automated and is redundant. And having a robot or an algorithm run, taking information from one to the other, they’re able to do it-it’s able to do it-it’s and it, you know-much faster, number one, but also with fewer errors. However, the person or the human in the middle of that equation is to ensure that it’s the right information at all times, number one, but it’s also relevant and useful information. So, it’s not that it’s replacing; it’s augmenting that process. And I think the fear comes, they are not understanding a lot of these new technologies, sort of the whole black box concept of what do natural language processing applications mean, is the chatbot actually listening? Are they providing the right information? And sort of understanding that, even from a developer perspective, as well as the user perspective?
Brian: Is this bringing of these new technologies into your organization, is this something where you’ve had to come up with a repeatable process to do this because it’s happening a lot? Or is it more-it’s infrequent that you’re going to stir the nest up a little bit with some of the things-
Besa: Oh, I have stirred the nest [laugh].
Brian: -quite a bit. So, do you have a repeated process for designing these experiences in so that they’re not as disruptive, and maybe there’s more adoption early on, there’s less resistance? Or is there a way you go about doing that?
Besa: I think earlier on, to sort of engage with the right stakeholders and the key stakeholders is really important. You’re going to have important gatekeepers, they’re going to say, “No, no, no.” The naysayers. I start with the naysayers first-the harder nuts to crack-and say, “How can this improve your process or your service?” If I could win them over, the rest of this cake, I think. Well, almost. Not all the time.
In a way, if I could show-that’s why I was mentioning smaller pilots-that if you could show smaller wins, that shows that outcome or demonstratable value, then it’s an easier sell across the board. But if you have nothing to show except for concepts and theory, nothing in real they could touch, then for them, it’s much-and for me, it’s a harder sell across the board to bring in an additional technology and say, “Hey, this will bring an additional value.” If I haven’t demonstrated a value for product number one, there’s no historical track that I’m able to demonstrate value for a product number two, if we bring on board, even though that might be analogous and supportive structures. I think it’s really important to have individuals right from the beginning co-develop application, the way we’re working with our vendors. I mean, we understand the process much better; we understand the environment much better.
They have the technology, but we have the environment that the technology needs to adapt to and meet the needs. Same thing for our users. I don’t understand specifically what their day-to-day looks like. But they are the key stakeholders in that process to help us co-design an application that actually meets their needs. So, their voice is really important in that process. So, having those voices earlier on just makes everything much easier across the board.
Brian: It was interesting, you talked about, you kind of go after the naysayers first, which I thought was interesting. I think a lot of places try to find the evangelists, find the advocates, and then kind of work it a different way. So, how much of this is a push versus pull? You’re pushing new tech, new ideas, new ways of doing stuff with data on to them, versus they’re saying, “We want a chatbot to do X. We want to collect digital temperature readings.” How much is it push versus pull? And when you said they’re a naysayer, is that ever on their projects where they’re the ones that requested it and they’re also the naysayer? Or is it usually when you’re introducing something?
Besa: It really depends on the scenario, Brian, but that’s really funny that you say that. I had introduced the chatbot about a year and a half ago, two years. And I’m like, look, it’s a Mercy chatbot, and can help patients navigate. And at that time, they looked at the application, they’re like, “Very interesting.” And it didn’t go anywhere because the organization was not ready.
They had no idea what I was talking about, number one, even though chatbots have existed in many different industries. But it didn’t feel important at that point in time, they didn’t understand the whole concept of it. And I think initially when I came in, in sort of thinking through what are the systems now? What’s out there in the marketplace? What has worked?
So, don’t touch what has worked, right? The point is, how can we improve it, even incrementally, to make lives better, and then start there with small wins. So, for us, as you said, targeting those naysayers from the beginning, they become your biggest evangelist, if you could show them, “Here, look. I have improved a service or value for you, and I’ve saved this much time, and this is what your outcomes look like.” In a way, you don’t have to have those evangelists beating the drum. You have the hard naysayers kind of saying, “Hey, this worked for me,” actually opening those gateways and saying bringing the additional technology, bringing in this additional resources to help us.
Brian: Got it, got it, I didn’t really give you a chance to talk about exactly what it is that your organization does, and also who are the literal end users of these different digital solutions that use data, whether it’s internal, or patients, or whatever. Can you give people an overview just so we can put the show a little bit more into context.
Besa: So, our end-user is usually our social workers, our mental health counselors, our pediatricians, our nurses, our psychiatric nurse practitioners, our healthcare managers. So, it’s basically healthcare providers that are out there in the frontlines providing services to the most disenfranchised communities in New York City and Long Island. So, these are the people, during time with COVID, they’re out there visiting families providing those basic support needs; especially now, we see how important it is, and providing them with the right tools, providing with the right technology to be able to do their job safely, securely. Especially early on during the pandemic, we didn’t know what this would mean, as far as infection arrayed within communities, and my team was able to take a lot of the work that Johns Hopkins did and use those mapping to figure out what our service provision areas look like, as far as infection rates. And then figure out a plan, where can we deploy staff into the field as safely as possible to meet patient needs, obviously because that’s the most important thing that we do provide is that home-based, community-based services, but also do it in a safe way so our frontline workers or our essential workers are not affected by this pandemic.
So, those are our end-users have a lot of the technology and the products that we have developed. And we moved earlier on to move to online and virtual platforms, without thinking that pandemic was looming, we had no idea, but we’re better pressed to deal with some of the challenges that came afterwards because we had done a lot of that work earlier on.
Brian: Mm-hm. We were talking about in our chat about human-centered design in the context of these data products, and you mentioned this co-design activity, where we have, I assume, stakeholders or literal customers are involved early and along the way; they’re not waiting three months, and then seeing something for the first time that they’d never heard about. Is that working for you? If it’s not working, what are the challenges involved with that, with actually getting them involved and kind of brought along the way?
Besa: It’s definitely working; there’s always challenges. One of the biggest challenge is time. They do have day jobs, they’re committed to-I’m asking them to sort of comment or provide support into products and services in addition to what they have to do in their daily nine to five. So, time has always been a challenge. But like I said previously, if you could demonstrate that their added time and value will bring a resource that would save them time, then we have quite a lot of buy-in.
But I think human-centered design is so critical for any technology and product that will be developed in the next 10 or 15 years. And the more that it’s integrated into our daily lives, where it’s an afterthought. I mean, you have, like, Siri, or Alexa in the background and I could just ask them what is the weather’s like without having to log on my computer. It’s being integrated into the way we’re doing things, makes things much easier to adopt technology, even for specialized services. Let’s say-I mean, we have kids with autism that we provide services.
So, utilizing technologies to help improve their health and well being, number one, but also in a way that allows them to function in their environment, with technology support, so they’re able to lead almost normal lives with these supportive resources. I think that’s where the future is going to be.
Brian: Mm-hm. And how did you bring this skill of doing human-centered design into your group? Do you have designers and UX professionals that lead that, or is it something where everyone does some of that? Or who runs that process?
Besa: So, we partner with external organizations, and we’ve done a lot of seminars, other things where we learned about human-centered design, but also working with our vendors and partners to build that into the project portfolio from the get-go rather than sort of as an afterthought: “Oh, we have to think about how this will be integrated into the environment.” I think thinking about the integration piece, thinking about the environment from the beginning, helps sort of have a better fit between a product and a use of that product within that process, or that environment. So, working with our EHR vendors, our AI vendors, our different sort of technology companies, and having them work alongside us. So, as far as having skilled folks from their divisions because we don’t have the capacity to have UX/UI folks that-or engineers, you know, to have on board. And for us, it’s a cost that I cannot consume or maintain, but bringing in that for every single product portfolio, requesting that from our vendors. I think that has been what we have done.
Brian: Have you seen a difference in how the outcomes are when you do and you don’t do that? Or was there a moment where you’re like, “Okay, in the future, I’m doing it this way because this other way didn’t work?” I don’t think that’s how a lot of-especially in the data products world-I don’t think that’s how a lot of places do it. And so I’m-
Besa: They don’t.
Brian: -curious, what made you decide that we need to do it this way? What was the moment or was there a particular-
Besa: Failure?
Brian: Was there a failure? Or was it one, was it a build-up? And how did, like, this is what we need to be doing to avoid that problem. Where did that come from?
Besa: Well, like I said, failure. Initially, we had designed a COVID app about a year ago, and this is a demonstrable failure, where we had designed it, we brought another product, other partner, and decided to deploy it, and realized pretty quickly that the questions that we were asking, the fit with the environment, the fit with our users, was definitely way off, and learned pretty quickly that I got quite a lot of resistance across the board-my team got quite a lot of resistance. And we realized pretty quickly that we had to scrap it even though it was like two or three months worth the work. Iteration number two, we started early, involving the different stakeholders and having them tell us, you know, what are the essential components? What do we need to track?
How do we need to track it? How do you want the information fed back to you? Because they’re the consumers of that information. And building those pathways in, definitely, we saw a change, also buy-in, and not only the development, but also now that we’re finally rolling out the application that we developed with our Microsoft Partners, that the adoption was much greater, the ease of that adoption because it kind of fit, and then we took their questions, their comments, and concerns into that development as well. So, failure is usually mother of all inventions, as I think-or necessity, I’m not sure. [laugh]. One or the other.
Brian: Yeah, yeah. That’s how a lot of places I think, figure it out. It’s like, [laugh] all that development doesn’t count if no one uses it when it gets out. It just doesn’t matter how great the model was, or whatever the underlying technology is-
Besa: It was a dud. [laugh].
Brian: -yeah, if it doesn’t get used, it doesn’t matter. So, this is related to that question, but as you move into starting to use artificial intelligence, and machine learning, and some of these newer technologies-and maybe those are vendor tools-is anything now changing, again, about how you approach doing that because they use these different technologies? Or not necessarily? Is it mostly the same?
Besa: I would say it’s slightly different. I think with AI, there’s still that fear, even though AI has been around; if you look underneath the hood, it’s not anything too earth-shattering. It’s just scale. Most of the algorithms have been around for many, many years, it’s just a scale or new approaches of integrating different methodologies that especially now, within the development cycle, we have seen with neural nets and few other things. So, there’s still that fear as far as how the information will get used, what information will get [00:32:08 unintelligible]?
I think, for our clinician, one of the questions that one of the clinicians raised is that, “Why would I trust an AI telling me this patient needs this? I have 20 years of medical school.” And what, basically, I told him, it’s not that I want to replace you, or your medical school, or your clinical training, I’m just augmenting and sourcing another level of information that otherwise it might take you a long time to source, or we all have blind spots. So, basically, just like with a GPS that tells you, “Go this way, this is the shortest route.” I’m just showing you another alternative options. But it’s up to you and your clinical decision-making to say that’s the right option, or that’s not the right option. Is just sourcing or augmenting that information to allow for better decision-making. But there’s still fear around that for a lot of folks.
Brian: Are you finding yourself having to change the product or the experience to accommodate that? Or is it, wait for the people to change? What’s your approach?
Besa: I think, changing the product. So, we were working on a natural language processing application with another vendor. And we’re co-designing that application. I was basically using our EHR narrative data to source different information that otherwise our clinicians, it would take days or weeks to actually look at a patient’s medical records-and some of those medical records can be years old-and basically show them a timeline of what it looked like within different domains of health, well-being, mental health, that it’s an additional layer of information, including what are the crisis points that the patient had throughout their timeline. And for the clinician to have that displayed in a way that they’re able to consume that information.
So, we had to work with them in that process to design and include the right elements, not necessarily as far as the sourcing of their information, but more in the consuming of that information. How do you want this displayed? How do you want this visualized? What would be important for you to be alerted at what point in time? And how will you use this information? I think that’s the most important components that we have gotten a feedback and designed that from the earlier on, with our providers.
Brian: And do you guys have a process for validating those designs and experiences prior to a full technology commitment to building them out? Like, at the mock-up stage, the prototype stage, do you do some kind of work, or not so much?
Besa: No, we do work throughout the process, asking feedback, focus groups with our providers: what they like, what they didn’t like, what they think needs further development as an iterative sprint process, where there’s changes with different information coming rather than just waiting for the product to get developed and say, “Hey, here you go.” And then you’re running into continuous reiterations. I think it’s just easier to reiterate and configure the product earlier on, so that it fits, rather than just having something off the shelf, and trying to make it fit.
Brian: Yeah, yeah, yeah. Is it difficult to get-I mean, you talked about this time, so I’m always interested in the ability of people to provide the fee-even though it’s in their interest, right, it’s getting that access to their time and all of that. Do you have a specific team or people that kind of own that process of getting it in front, doing evaluation, taking the learnings back to the engineering, or data science team, or whoever is involved, is that owned by one person, or maybe the partner owns it.
Besa: So, my division I was at, that’s part of the evaluation aspect of it, working with our vendors, sort of soliciting that continuous feedback from our end-users, and then taking that feedback back to our vendors and customizing it further. So, we have to own it, somebody has to own the data, the analytics, sort of the tailoring of those applications, but I see that as almost like a partnership relationship. I’m not alone; I’m in an ecosystem with others. And I see our partners, our vendors, our providers as being co-developers, and a lot of these applications. I don’t say that I’ve developed them: we have done this, not I have done this.
Brian: Mm-hm. Is that difficult? Just, I would think the dance of what they-you know, they have their product roadmap. You guys need it to be, you know, red, they, like, “Well, blue, we don’t really have a way to like-” and I’m oversimplifying talking about theming colors here. That’s not the point. Is that a difficult thing to navigate with them and constant trade-offs in this kind of thing?
Besa: It’s always a difficult thing with our partners to navigate. They have a certain product roadmap that they need to stick with. And I want development a lot faster. So, what my team has done is created workgroups with our vendors and others to sort of shift developmental timelines, number one, but change there sort of roadmap as far as what needs to go into production first-or development and production first-and then ensure there’s a tiered approach to meet our needs, or meet our partner needs because we work as a collective. It’s not just one healthcare organization: there’s many health and social service organizations that are on the same boat. So, having our partners be responsive and reflective of what we need, and have their product portfolios get tailored to, for us to consume them, I think that’s important on their part.
Brian: Sounds almost like the unions, you know? [laugh].
Besa: I know, wouldn’t it be great to go back to that time-
Brian: [laugh]. Solidarity.
Besa: [00:37:27 crosstalk] collective? [laugh].
Brian: [laugh]. Well, Besa, this has been a great conversation. Just kind of in closing, I wanted to give you the microphone, and a chance to tell us anything else that you think my audience needs to hear about in terms of how you design human-centered data products and driving adoption, making them more valuable and useful. Do you have any other closing advice or things you’d like to share?
Besa: Sure. I mean, I think for me-and this is sort of lessons learned-is that it’s really important to think about the human in the middle of this entire process. That sometimes products get developed, things get developed, and there’s a next shiny object without really thinking about, do I really need this other shiny object in my life? Is it going to improve the way I do things? Is it going to improve anything?
Then why do I need this additional shiny object? Same thing, like earlier on, I bought an iPhone and a Samsung, and then I ended up downloading a whole bunch of different applications, and I’m like, “I need this, I need this, I need this, I need this.” And realize pretty quickly that I only use a few of all of those applications. I mean, those applications were better tailored to me, number one, but also build tailored to what I needed at that time. So, when I think about personalized medicine, personalized health, personalized anything, the more personalized a product is, and a technology product is, I think the better it is and the greater the adoption.
Brian: Preaching to the choir here on that. So, thank you for coming on and sharing your experience with us. It’s been great to have you.
Besa: It’s a pleasure, Brian, as usual. Thank you.
Brian: Yeah, thanks.
Ahmer Inam considers himself an evangelist of data science who’s been “doing data science since before it was called data science. With more than...
Julie Yoo is the co-founder of Kyruus, a medical technology company that is the developer of ProviderMatch. One of the most frustrating things about...
Dr. Bob Hayes, will be the first to tell you that he’s a dataphile. Ever since he took a stats course in college in...
