Helping leaders make effective decisions through the use of data
Epidemic X
Epistemix is a start-up that helps different types of leaders, policy makers and strategists make more informed and impactful epidemiological decisions through the use of simulation data. They do this through the use of a powerful modeling tool and language called FRED (Frameworks for Reconstruction Epidemiological Dynamics). FRED can model disease spread by simulating interactions between the people in a certain geographical location using synthetic population data and agent based modeling (ABM). Although it is very powerful, FRED hasn’t been able to grow into its full potential because it is highly technical and runs on a command line interface, restricting the number of users it can have. The output of these models were also a barrier to its use because it came in thousands of rows of CSV files of simulation data of different potential scenarios and it wasn’t digestible.
Executive Summary
Our original goal was to increase the number of users that FRED has by making the tool easier to create models. However, as the business goals of Epistemix changed, our project changed as well. Epistemix no longer wanted us to focus on improving the modeling process but they wanted us to focus on helping users make sense of the thousands of rows of CSV data that is produced by the simulation. Our final solution addresses the concerns of leaders who are trying to explore tons of simulation output data but also dive deeper and use this data to make effective decisions.
Team: Worked with a team of researchers, a visual designer and a UX designer
My Role: I worked on the team as a Design Strategist who supported the work of both the designers and researchers on the team and aligned the goals of both research and design. I also was an Interaction Designer on the team that worked to bring the research insights to life.
Timeline: January to August 2022
Skills: User Research, Wireframing, Storyboarding, Rapid Prototyping, Figma Prototyping, Photoshop, After Effects
Who We Are
Epistemix currently focuses its solution around FRED - a powerful modeling platform that requires a local installation process. As an organization that runs on a smaller scale it offers personalized one on one help to their clients, often acting as a personal troubleshooting system for errors.
Users who use FRED have experience with programming in some form. Epistemix now wishes to expand FRED’s user base by designing a system that allows people to explore the power of agent based modeling (ABM) without getting into the intricacies of code. This would mean expanding the domain applicability beyond public health as well as lowering the barrier to entry to allow more non-expert users to pick up FRED.
Original Problem Context
Actual FRED code used to make a model
Research Goals
What are the biggest challenges that modelers face when modeling with FRED?
What do modelers wish their modeling workflows could have to make it a better experience?
How can we make FRED more usable for people with varying programming abilities?
Target User
In our research, we focused on an epidemiologist who has limited programming knowledge. We chose the public health domain because we had the greatest access to interview participants in or analogous to this domain. Additionally, it is a domain that Epistemix is already familiar with so it made feedback sessions and communication much easier.
Epistemix came to us with the idea that they wanted a visual coding platform that would turn the FRED language into blocks or boxes of some sort because to help solve the problem of the language being too technical and gaining non-technical users. Although this is a great product and suggestion for our project, we understood that we needed to complete research to figure out if visual coding is actually what the users want and need. Epistemix adamant about creating a visual coding platform, therefore we needed to figure out a way to balance the needs of the user and the client.
Balancing the Needs of the Users and the Needs of the Client
To ensure that we were able to explore the problem from a broader perspective without losing sight of Epistemix’s product goals, we decided to follow a parallel research approach as follows:
In one parallel, we would focus on broad, generative research, which would focus on the overarching needs and pain points that modelers face today.
In the other parallel, we would focus on the possibility of a visual coding interface that would allow modelers to build models through a GUI, thus eliminating the need for in-depth programming knowledge.
This approach provided us with the ability to explore the problem space holistically, without being zoned into a single solution space from the very beginning.
Loose interviews
Goal: To get an initial understanding of how predictive modelers go through their workflow, we wanted to loosely interview modelers to gain a sense of their general attitudes.
Who: 6 modelers with experience in various predictive modeling domains specifically, epidemiology, machine learning, and networks
How: These took on the format of moderated research conversations, where we informally asked modelers questions about their modeling process and allowed the conversation to develop with little structure.
Contextual Interviews via Think-Aloud Interviews
Goal: To understand users’ pain in working with FRED, handling errors, and creating motivation.
Who: 5 FRED experts (3 internal and 2 external)
How: We conducted In-depth observational studies and follow-up interviews on users’ behaviors when using FRED.
Methods
Using the findings from our loose interviews and our contextual interviews, I created an empathy map to gain a deeper insight into who the users are and the issues they are facing and portray that to the client.
Artifact Analysis/Individual Journey Mapping
Goal: To holistically understand the entire workflow experience of Agent-based modelers
Who: 3 ABM modelers (non-FRED users)
How: We asked them to provide detailed insight into their goals, tasks, and emotions for each stage of their modeling process
Pretotype (Pre-prototype to determine what concepts to design)
Goal: To understand the visual mental models of FRED users and agent-based modelers
Who: 3 FRED expert modelers (internal to Epistemix)
How: We gave the users a short story that follows a family in the fictitious Wellington City as they interact with Covid-19. We asked users to read the story, draw a picture of what they just read, and then draw a picture of what they think the simulation will look like for the subsequent 2 days. We also asked participants to verbalize their thought processes while drawing.
After these 4 research activities, I consolidated our data and created an aggregate journey map to help us understand the aggregate modeling journey.
Research Insights
Building a model from scratch is a pain because a lot of time is expended on setup rather than on the scenario-specific problems that modelers are hired for working on. Users long for pre-made model building blocks, which will ease the burden of model environment configuration and thus speed up the modeling process.
Modelers prefer sacrificing control during the set-up and foundational stages to surpass technical difficulties, as long as they still have power over the agent interactions in later stages of the modeling process.
Prioritizing quick wins like improving the setup, documentation, and error resolution strategies are essential for retaining the current customer base until a new FRED experience comes into existence. Presently, the need for FRED is outweighed by the frustration caused by these factors.
FRED modelers wish for an open community where they can share their problems, get answers to questions, pose discussions, and get modeling inspiration from other fellow FRED modelers. Currently, the only channel for support and communication is via email and messaging with Epistemix employees, which is cumbersome and unsustainable for both parties.
Early Stage Design
When it came to testing product concepts and ideas we ran an activity with 4 participants who had FRED modeling experience in which we asked them to provide thoughts on the storyboards (sketches of FRED scenario-specific solutions) we presented to them. The activity demonstrated early operation issues internal with FRED along with a focused need on templates and a community-oriented support system. I focused specifically on creating a solution that allowed users to toggle between visual and text coding; the visual being easier to digest but the written code still gives the user more control and flexibility.
As we were narrowing down on which concepts the research was telling us to target, Epistemix approached the team and told us that business needs of the company are changing. Because it is a start-up, they need to act fast and be agile in order to meet the demand. In turn, we needed to be agile as well and adapt quickly to this new business venture they were looking into.
Project Pivot
Epistemix no longer wanted us to focus on making the FRED tool and language easier to use but now they wanted us to focus on the output of the FRED models. In the current state, the output of the models are hundreds of CSV files each with hundreds or thousands of rows of data and it puts the burden on the user to make sense of all that data and create their own visualizations. This data is important to be able to understand because it is of many different scenarios or potential futures based on the simulation input parameters that were used to create the model. Each scenario is a potential future that could be analyzed in order for a user to make a decision based on their specific circumstances.
New Problem Context
Because of this new problem space, we were looking into a new target user as well. We were no longer looking at an epidemiologist with low coding experience, but we were now looking at a policy maker who is trying to use the data from the FRED models to make important decisions that could impact the lives of their citizens. This could potentially be on the city, county or state level.
Because of our shorter timeline, we decided to narrow down a specific target user with our client rather than starting generative research to narrow down a user. Our assumption was that it would be more beneficial for us to dive deep into one specific user and use case and expand the product from there rather than trying to design for many use cases in the beginning.
New User
A public health planner working within the infectious disease unit within a regional health authority to do an initial risk assessment in response to reports of an infectious disease outbreak in their area and develop a recommendation for how to respond.
With this use case in mind, we created a persona to advocate for and show the importance of focusing on the user to our client.
Our use case is at follows:
Isabella works at the Allegheny County Health Department as a middle management public health planner. It’s December 2020 and another potential spike of COVID-19 is of rising concern. She’s been assigned to lead a team in simulating how COVID-19 might develop in the next five months and what that means for policy recommendations regarding lockdowns and stay at home mandates.
In order to get the team and the client realigned on our goals for the rest of the project, I created a project plan with weekly goals and reflection points. This allowed us to set expectations each week and reflect on why/how we reached or did not reach those expectations. This also fostered effective communication within the team and with our client.
Project Plan
We started generative need-finding research again because we are entering a new problem space. To understand how policymakers currently interact with data visualization interfaces, we conducted semi-structured interviews. These interviews were mainly conducted by having candid conversations with different people in policy to understand the needs and pain that they face when working with current tools in their workflows.
The major conclusion was the need for temporal, spatial and comparative data, along with the ability to compare scenarios with one another.
Need Finding Research
Because the deadline for our project was quickly approaching, we needed to make some assumptions before we started designing.
Assumptions
That a web-based interface is the best solution for this product. This is due to the resources that Epistemix has and how they interact with customers.
Creating visualizations for the policymakers is the best way to help with their analysis and understanding.
The policymakers are either already used to doing this type of analysis digitally or will be able to learn how to do it digitally.
The public health planner is a middle management policy maker that would usually suggest 2-5 scenarios to upper management when important decisions are being deliberated.
Narrowing Down
Another method we had to use a lot due to limited time was crazy 8s and crazy 5s to get solution concepts out there. We often would do this with the whole team, our client and different advisors. We would validate them with our client and see if the team's ideas matched with the business goals of the client.
After 3 rounds of sketching and validation, I created a simple workflow model for the team to follow to illustrate how our product should help the public health planner explore the large amounts of output scenarios but still narrow down and select the few scenarios that they would want to show to their upper management.
This model is similar to half of the double diamond method, where the user starts with shallow convergence and they approach our solution with a basic idea of what they want to explore. Then the user moves on to diverge and explore all the potential scenarios that the solution could offer. Finally the user would converge again, this time more informed due to their explorations. They will be able to narrow down to specific scenarios of interest that they can share with their colleagues and upper management.
Half Diamond Workflow
I conducted a competitive analysis not only to understand the competitive landscape, but to also understand some common struggles and gain inspiration from other solutions. I researched over 10 different competitors that focused on modeling the future and uncertainty and gathered basic information about each in order to use as inspiration or figure out potential advantages. However, I decided to dive deeper into 4 competitors that focus on using simulation data focused in either epidemiology or a policy related field.
Main Findings:
Although there are benefits to both, most products either have too much or too little information on the screen at once, making it difficult for the users to do their analysis.
Most products force the user to collaborate outside of the tool (e.g. take screenshot and share on Slack with descriptions)
Competitive Analysis
Designing
When it came to setting the goals for our designs, I took into consideration our need-finding research, our assumptions, and results from the competitive analysis in. I outlined goals that help address Isabella’s needs and objectives using our product.
Design Goals
How might we help Isabella find important information to look at?
How might we help Isabella asynchronously collaborate with her teammates?
How can Isabella explore and figure out what to focus on amongst large amounts of data?
The biggest thing that I wanted to focus on was balancing the ability for Isabella to see the breadth of potential scenarios and the ability to narrow down on a select few scenarios and make effective decisions with her team.
Introducing Epidemic X
When it came to developing features, the team kept the design goals in the center of our design process and iterated based on these goals.
Smart suggestions nudge Isabella to look at relevant information scraped from online that she may not have found herself. These help her contextualize the data. These smart suggestions come in the form of historical comparison and the ability to show the lowest vs. highest cases.
Design Goal 1: How might we help Isabella find important information to look at?
One of the ways the Epidemic X system suggests content is through pattern analysis to match simulated scenarios to a historical database of diseases. Isabella is familiar with many case studies of previous diseases. However, it’s hard to rely on her memory of all the patterns she has seen in other cases and match them to the simulations produced by Epistemix. The pattern analysis feature allows Isabella to easily draw comparisons to real case studies so she doesn’t have to search for them herself.
Feature: Historical Comparison
Throughout her work, Isabella has to share and explain her findings, and sometimes develop an argument. Epidemic X provides suggested starting points for investigation. In the grid view of the interface we show lowest and case scenarios uptop. Because Isabella has chosen to view the cases by infection count, the best and worst cases are represented as the highest and lowest infection counts. This helps Isabella build a fuller argument to share with her colleagues because she can explore both easily.
Feature: Lowest vs. Highest Cases
Most of Isabella’s work relies on in-person discussions, but she still tends to do work on her own before sharing with colleagues. Now, she can share access to her work in Epidemic X with other colleagues to make comments and export graphs.
Design Goal 2: How might we help Isabella collaborate with her teammates?
Comments can either be made about the overall page (global comments) or they can be placed on specific points in a graph (in-context comments).
I derived inspiration for commenting from Google Docs and their sharing permissions, Figma’s in context comments, and social media like Facebook’s ability to make overall comments on a post.
Feature: Commenting
Global Comments
In-context Comments
Isabella also has the option of exporting any graph via email or to social media.
We received input from user testing that the ability to share individual graphs is very powerful. Exporting graphs allows Isabella to send snippets of her work to colleagues. This makes the data analysis more digestible.
Feature: Exporting
Design Goal 3: How can Isabella explore and figure out what to focus on amongst large amounts of data?
Epidemic X promotes exploration by modeling our interface’s flow similar to that of online shopping (e-commerce). This helps Isabella understand that simulations are not quick and easy predictions, but rather complex and require exploration before any consensus is reached.
Data viz interfaces are often too hard to use because there is either too much or too little information to do anything. The e-commerce flow helps her consume large amounts of data in a systematic way with filters and different ways to view the visualizations (such as the main view and grid view).
The first half of the user flow allows the user to explore by filtering and receiving suggestions for focused exploration, just like an e-commerce site allows.
Feature: Filtering
The second half of the journey focuses on narrowing the exploration with bookmarks, similar to the shopping cart.
We derived inspiration from the e-commerce shopping experience. All e-commerce sites allow for filters and sorting to help view the large amounts of products in an easier format. We also created a bookmark page that mirrors the shopping cart experience, where the user can evaluate the products they liked before deciding to purchase (or in our case, export).
Feature: Bookmarking
The value of Epidemic X exists in the ability to translate the data of our simulation into an interface where various futures are explored. It allows users to validate and explore their hypotheses by demonstrating potential futures in an e-commerce inspired flow, which is something that is already familiar to them.
Our Competitive Advantage
Comparative analysis
Collaboration
System suggestions
Most interfaces are not able to capture the power of information in terms of breadth and depth like Epidemic X.
Impact
The team saw our product having a beneficial impact on Epistemix and their potential customers. However, we recognized that more work will need to be done before Epistemix can implement Epidemic X fully. More evaluative user research will need to be done, leading to more future iterations. There will also need to be a feasibility study performed with developers to see what aspects of our design can be adjusted to the contemporary constraints of the company.
As Epistemix also expands their use cases for this product, there may need to be changes in our interface and visualizations as well. This is another place where user research may come into play. I created a product roadmap to outline these steps for Epistemix and guide them through design methods.
Looking into the Future
If I had more time on this project I would have liked to build another persona and discover ways our solution could reach that user as well. I would have also liked to complete research on some of the assumptions that we made and either validated or refuted them.
Overall, this project taught me a lot about design, myself and working with a client. In those 8 months, I learned how to handle ambiguity and be agile in order to come up with a solution that fit the needs of a user but also the business needs of the client. Our team needed to be agile and professional especially when working with a start-up, whose needs change often and rapidly. Another valuable skill that I was able to work on was advocating for the needs of the users with a client that isn’t used to doing that. My role also allowed me to teach our client about the importance of keeping the user at the center of technology design.
Because this was my longest team project and my first time working with a tech corporation as a client, I was able to work on my team and client dynamics. As the design strategist, I was able to implement team retrospectives and wellness diaries for each sprint. This helped us reflect on what we did well and how we can improve before going into the next sprint. It also helped the team camaraderie and make sure everyone was feeling strong as the project went on.
