Businesses need to spot risk quickly, but it’s easy for development issues to be missed until it’s too late.
While at XebiaLabs, we wanted to make DevOps easier to manage: we identified that spotting risk was key, and re-defined how we presented development in progress to make it easier to spot.
What do users need to succeed?
Soon after I joined XebiaLabs in December 2016, I participated in a design sprint looking to address the problem “finding potential problems in a release early.” The outcome of the week was a re-design of the existing XL Release overview screen that introduced the concept of risk to running releases. Each release would have a risk score that would change based on the current release status. We focused on some big release concerns we heard during our research day, when we spoke with user representatives including task due dates, failures, and impact to other releases. We validated the prototype with both internal user representatives and two local customers (Netherlands). After the design sprint week ended, my colleague Nadia and I collaborated across the Atlantic to carry the re-design through to reality.
The MVP feature was released in March 2017 as part of XL Release v6.2. Click here to read the official announcement, and here for the feature overview in our product documentation. After the release, as I was working on building our research program, we spoke to more current users upgrading to the new version and using the risk feature to get feedback on how risk worked in practice. The response was quite positive, with novice users happy to have some guidance on what needed attention, and expert users excited about the possibilities for integrating risk into their conversations around release management. We also discussed the risk feature in depth during the User Roundtable in June 2017. All of the feedback we gathered from these fed into a two day design workshop to expand on the initial feature, which I ran with my colleague in August 2017.
What does risk mean to users?
Once the MVP launched, we knew there was a lot of room to expand the automated risk identification capability, tailoring it to the specific ways our users define risk in their development pipelines. In August 2017 I worked with my Nadia to conduct a two-day design intensive focused on expanding the concept of risk in XL Release.
In the month before the design sprint, I reached out to our User Panel and coordinated several research sessions relating to the risk feature: some before the design workshop to support our understanding of how risk is used, and some after to validate the workshop outcome. As this workshop was shorter than a traditional Google design sprint, we were unable to have the time dedicated to research that would typically be included. Instead, I gathered risk feedback from the last few months and presented it to the workshop participants during common ground building on the morning of day one.
After the workshop I ran sessions with users to get their feedback on our design plan.The overall design was positively received, but I noted during validation sessions that users were confused about how risk was calculated, and unsure exactly how their changes would impact the overall risk score. After discussion of my findings, my colleague worked to refine the visual design, clarifying the relationship between each individual risk factor and the overall score thresholds. I conducted a second round of validation internally as a “usability party” where members of our Customer Success, Sales, and Implementation teams were invited to sign up for short slots with me throughout a day and test the design. This generated enough data to feel confident that the final designs would work for our expert users (the target audience for this feature), and be learnable for less experienced users.
The flexible, customizable v2.0 of the tool was released as part of XL Release 7.5, and is still a highly valued component of the product. You can read more at the XebiaLabs website.