The fun challenge of enterprise product research

Phew, I've been off the plane for a weekend and between jetlag and catching up with work I'm exhausted. But at least I'm back!

Every so often, someone asks me what types of products I like working on more, consumer or enterprise products? For the past decade or so now, my answer has been pretty consistent – having done both for large stretches of time, I really like working on enterprise products more. The primary reason is because enterprise products constantly offer new and frustrating challenges every step of the way.

That's not to say that consumer product research itself is not full of new and interesting problems. There's always new products coming out, new markets to explore, new consumer needs that need understanding. For example, I had spoken to a researcher that had been studying autonomous taxis and the various trust and safety issues surrounding that whole space. How can you learn to what extent people do or do not trust such systems? How does all that depend on the context and situation? How can you study all of that safely and ethically? How can you learn all this with prototype level equipment so you don't invest in a giant fleet that everyone hates? Super hard questions in a space that's very new, where few people, if any, has done work on. (And anyone who's done similar work is likely at a competitor and they aren't sharing notes). Even if you're not one of the handful of researchers working on some completely brand new market, there's still challenging and satisfying work to do.

But my excitement at working on enterprise products is because the whole space is... perversely challenging... even for some of the most simple questions. Mass consumer products, for all their their complex challenges, usually have very straightforward business propositions – you either satisfy a customer's needs enough that they are willing to give you money, or you go out of business trying. User satisfaction, at least within a competitive landscape, has a fairly direct causal path to success. (For the sake of simplicity I'm going to gloss over the many complicated bits that stray from this like monopolies). Enterprise products don't have nearly as strong a causal connection between user satisfaction and business success. Just look at the many people who complain about work software like MS Teams but are forced to use it for work.

Just about all the easy assumptions you can make within consumer products may not apply to enterprise ones. You can't assume that people use the product because they want to. Nor can you assume that even if people use your product and you have terabytes of telemetry, that it can tell you whether they like it or find it useful. Just think back to all the software you use at work that you absolutely detest. This calls into question a lot of the deep assumptions we as data analysts make about usage telemetry. Are thousands of users taking 20-click paths to do a task because they want to, or because we left them no choice?

You can't assume that there's more than a handful of people in the entire world that needs a specific feature you're building. If there's literally only five users for something, then are quantitative methods even the correct method to use to understand those users? Even if you get lucky and have "hundreds" of users, that tiny population lets you use some quantitative methods, but with a lot of caveats.

You can't assume that having a good user experience necessarily means someone is willing to buy or use your product because the person writing the check is usually not the end user. Maybe they'll switch to a competitor if enough people complain about it, but between switching and re-training costs, it takes a huge amount of screaming to get companies to switch software that is integrated deep into their business processes.

There's definitely some enterprise/B2B type software that has a big enough market and enough real competition where things feel a bit more "consumer-y". The example that most readily comes to mind is the point-of-sale hardware and software that small businesses now use to collect payments. Thanks to tech lowering the cost of terminals and transaction processing, the space went from having expensive proprietary terminal devices bolted to a cash register to effectively mini iPads attached to card readers. And those new devices have much more user friendly controls because competition has made the cost of switching to a less aggravating vendor much lower now.

But anyways, back to why I like working on enterprise products. The work demands a huge amount of careful thinking about what exactly we're trying to accomplish, because easy assumptions to build into our models are rare. Simple routine tasks like running an A/B test become thorny because maybe we won't ever have a sample size big enough to find the difference we're expecting – so what do we do instead?

Even something as simple as interpreting telemetry becomes harder. One story I often tell to others is how a friend of mine worked on measuring the effectiveness of an internal tool. The metric the team eventually decided to target was "time on page", they were going to make a bunch of studies and improvements and bring that number down. Then, after many months of work the team released a new version where individual tests showed that time needed to complete a task on that page was lower. The team launched their new version, qualitative feedback was that things had improved... but the team's target metric of time on page went up. It took talking to users afterwards to learn that while the time needed per task had gone down, the users had a busy queue of work to do so they simply moved on to their next work task.

While there's many different lessons you can take from that story, the one I think most about is whether it is even possible to predict the direction of a metric under enterprise conditions? I certainly was not expecting the outcome of my friend's story, and honestly no one else on the team was either. Most metrics can be interpreted multiple ways and it's really hard to come up with a convincing argument as to which interpretation is best. For example, if I'm in the business of making e-commerce software, is it good or bad that 20% of my user's time using the entire product is spent on the "returns" section of the software? Without knowing anything else, you could imagine situations where that can be a good thing, or a bad thing. The only way forward is to reach for other methods, like interviews, to collect different kinds of data in order to make sense of things.

Far too often in data science, we learn to "pull insights from data" by making hypotheses and assumptions about whatever it is we're analyzing. We see trends and relationships in the data, we know facts about how the data is collected, and then develop hypotheses about what is going on based on that information. By knowing very intimately how we instrument our web signup form and at what step users stop interacting, I can usually hypothesize reasons for why they don't proceed. Users don't like pulling out their credit cards. Users were not serious about signing up if they don't even want to put in their email. Users are put off by the shipping costs despite putting items in their carts. I very often say these things while looking at data because I've got years of experience looking at that data and they're useful rules of thumb. I can even triangulate such findings with other data points and even qualitative data.

But let's be frank here, all those statements are ultimately pulled from my imagination. Find a hundred users who abandon at one point in any flow and you'll likely find dozens of unique answers as to why they stopped at that point, including things like "my kid came to ask me a question and I got distracted". In consumer products where there are thousands, millions, even billions of users, a data scientist can get away with such conjectures because on average a large chunk of the population will experience very similar issues. But in the much more constrained world of enterprise users that is not necessarily the case. At the least, you need to dig deeper in order to understand what is going on.

Ultimately, this challenge is why I find enterprise work so rewarding. Nothing's ever easy despite the problems often being familiar "easy" ones. It makes for lots of room to experiment and be creative in coming up with ways to come to an understanding of what is going on.