Why

As business becomes more digital, there is a growing perception that high pace software delivery strongly correlates to high-performance organizations. Furthermore, references such as the State of DevOps Report have demonstrated it by using direct data and the scientific method. The continuous delivery in a broader sense, including the lean startup, is a powerful tool to support new product development. Creating a repeatable framework is key to a startup studio intending to build multiple products and businesses.

I will describe a system that evolved over the six months of building Tutorean.

The MV* mindset

Limited teams and low budgets call for simpler options out of sheer necessity. Even if this sounds reasonable, down the road, emotions surface, insecurity, perfectionism, and risk aversion bring complexity to the playground.

The concept of Minimum Viable Product refers to the smallest set of features of a product or service that spurs interest in early adopters. Giving birth to an even more actionable concept, the Minimum Viable Test or Experiment.

To offer a practical example, the product we were building was a tutoring marketplace, which had an assumption on the availability of tutors as service providers. Without a product yet, we created a landing page and started signing up and onboarding tutors. After a couple of months and thousands of enrolled tutors, we had validated a key business assumption.

Because assumptions are the cornerstone of the business canvas, the goal is to push the venture idea as far as possible before building it.

The Minimum Viable concept applies to anything, either it is a product, an experiment, a team, a brand, a feature, a process, etc.

Deployment automation

Deployment automation is central to continuous delivery. The decision to invest in automation is counter-intuitive. Moreover, it is possible to execute experiments manually, with little effort or overhead.

The Lean elimination of waste principle, recommends delaying activities until they are necessary. Still, the ability to iterate fast, shipping experiments and features, was instrumental and required deployment automation to succeed.

In our scenario, deployment automation included:

  • Cloud-native applications based on IaaS;
  • CI pipeline with steps such as linting, code build, test execution, container build, upload and run;
  • Data migrations as needed;
  • Container versioning as a proxy for module and application versioning;
  • Single-click deployment by any team member;
  • Three similar environments: development, staging, and production;
  • Secret and configuration management per environment;

Scrumban and Sprint-less

Initiated with a loose Agile, once the fast deployment frequency kicked in, the concept of sprint became hollow and lost its meaning.

The sprint was no longer a unit of delivery. Thus, releases were abolished and numerous deliveries to production took place every day.

Even though there was a weekly planning meeting, iterations and scope changes happened as needed.

Also, the team had loose roles. Anyone could lead the planning or the daily stand-up meetings.

Retrospectives occurred monthly and remained relevant to the continuous improvement process. They were effective in defusing team stress by sharing pain points and promoting improvement plans.

Estimate-less

The team started estimating with story points using the planning poker exercise. There were plenty of discussions since the perception of task effort was not shared. After a few weeks, we moved into story points applied to tasks to improve the perception of task effort. A more peaceful but lengthier process took place.

The team had extensive experience with estimations, before the story points, there was t-shirt sizes, functions points, effort using expert judgment and effort by similarity, budget-driven, time-boxed development, etc…

At this point, the team was:

  • Splitting stories into smaller and meaningful tasks owned by a single person and delivered to production in a short time;
  • Spending half a day per 4-day week on estimations (12.5%);

The conclusion was the replacement of the estimation process of granular tasks, by an average effort of 1 story point, to remove waste but still de-risk execution. Gaining back extra time to build the product.

Visualize work & limit WIP

This is a simple concept but I cannot emphasize enough how powerful the ability to visualize work is.

It is an effective way to:

  • Communicate the state of the product, process, and inventory;
  • Visually coordinate tasks and establish dependencies;
  • Check who is doing what now;
  • Identify bottlenecks and reduce WIP;

According to Lean, WIP is a plague that needs to be minimized, due to:

  • Context switching cost;
  • People in multiple projects/tasks at the same time lead to further communication and coordination needs (meeting hell);
  • It pushes for rework and duplicates efforts;
  • Adds hand-off delays;

An effective practice included limiting the amount of acceptable ongoing tasks per process stage, and the visualization of the troubled stages via a clear redness that screamed for action.

“Git Flow”-less

Starting with a traditional Git Flow, after a few weeks, it became clear that:

  • Work was piling up in the staging environment, as a result of having multiple features and tasks mingled together;
  • The developer work finished when the code was delivered to the test environment. Testing it was a second job for the other team members;
  • An eventual bug would force developers into late context switching and halt opportunities;
  • The all or nothing led to a delay in releases every time a single task faced critical issues;
  • Continuous delivery was seriously harmed;

So we introduced the following improvements:

  • Features, independent tasks, and bugs had branches that would be born and die in master;
  • Pull requests made to master (and staging) to support the code review and the deployments;
  • Clearer ticket ownership, the developer held the responsibility to push the ticket to production, and in the process, seek pull request’s approval and functional validation from others;

No change is without shortcomings, but these improvements led to faster delivery and a sense of empowerment.

Test-less

Tests should be ubiquitous. Testing types exist to suit all tastes. Manual or automated. Black box or white boxing. Functional and non-functional. Usability and performance. Smoke or sanity.

Tests are undisputed as the key mechanism to enforce product quality. Furthermore, development practices were born around tests such as TDD and BDD.

But, tests also carry a short term heavy toll on the programmer for a long term benefit. Test cases need to keep evolving across iterations. In fast pace scenarios, there is a snowball effect. Yet, the long term benefits might not be realized as the future is unknown.

The subversive question here was: what is the minimal amount of tests needed before I run into troubles?

After iterating for a while, we reached the following set of practices:

  • Experienced staff with design ownership;
  • Extensive pull request review for side effect analysis, static code analysis and style guides included in the build process;
  • Minimalist pull requests, average between 5 and 50 lines of code changed;
  • Each ticket validated manually by someone else of the product team;
  • Delayed test automation until we had a significant negative impact on production.
  • Runtime error monitoring, unique errors generated slack messages and JIRA issues automatically, as a quality mitigation strategy.

At a certain moment, we had the sign-up unavailable for a few hours due to a configuration issue, which is a deal-breaker for any business. We decided to automate API tests for the key product functions, e.g. login, sign-up, session scheduling, and session purchase;

Warning: the typical matured user is not sympathetic to quality issues and automatic generation of tickets on runtime errors does not scale.

User Analytics

Expect sanitized answers while asking questions. Users tend to say something and do something else. User insights are a key aspect to support the validated learning process. This goes well beyond the conversion funnel.

Some of our strongest intuitions might be proven wrong by witnessing what the customer does. When we built the tutor sign-up and onboarding process, we assumed that the sign-up should be minimalist to avoid attrition, even if we needed much more data. Consequently, we created a gamified onboard process to obtain the data still missing, while explaining how the product would work. In the next weeks, we came to understand that the tutors were more than willing to provide all the data at first in a lengthy sign-up process, increasing immensely the likelihood that they would become service providers to the platform.

Building user analytics in a product is expensive, as well as unnecessary. Some tools are easy to integrate and offer a free tier. Try them and keep the more convenient ones.

Here is a subset of the tools we used:

  • Hotjar is a voyeur’s dream. Product owners and product marketing love the videos of the user interaction with mouse trails, heat maps, and clicks.

  • Mixpanel and Intercom are the solutions for google analytics refusal to hold user data. Opening the door for all sorts of data-driven marketing campaigns.

  • Google Tag Manager is a coder’s dream with a performance penalty. Include code once and let the marketeer promptly push configurations to intercept and track meaningful events.

  • Optimizely, Google Optimize, and Instapage facilitate running experiments and A/B tests with little intervention from the tech team.

Reflection day

This is a case of serendipity. For those paying attention, I mentioned a 4-day week. Due to the unavailability of some team members one day a week, we planned the week differently to reflect this constraint.

We ended up using Friday as a reflection day with the following activities taking place:

  • Ideation, market research, and business canvas as part of our venture innovation process;
  • Continuous improvement, re-thinking what we could do better to improve;
  • Product insight analysis and next week plan;
  • Personal projects;

We were able to turn a negative circumstance into a positive outcome.

Next ventures

The existence of a venture cycle creates the opportunity to do better in the next venture. The key improvements identified were:

Add a headless CMS to the stack. The continuous loop of copy tweaks, CTA A/B testing, and template change, scream for code-less and instantaneous content change.

More infrastructure as code. Having templates for building up clusters, VPCs, ALB, etc, would speed up the venture infrastructure bootstrap. Furthermore, AWS UI is a pain.

Improve tool integration and increase build automation. There are still manual tasks syncing the project management to the source control tool that can be further automated. Also, two pull requests per task is an overhead that could be avoided via implicit deployments to the test or staging environment.

A better hypothesis validation framework. Validated learning is harder than we anticipated. It is difficult not to keep second-guessing the results along the way. The data is not significant or the experience was not well-designed or the execution was ineffective or…

Monolith vs micro-services decision framework. Decoupling is a de-risk tactic to a long term strategy. Monolith supported by a framework is a short term quick win. The balance between both goals needs to be supported by explicit criteria.

Main takeaway

Being clueless about the product one needs to build is a get out of jail free card. Early-stage startups are corner cases that can fast track product development over slow-paced incumbent competitors.

What if any business could improve its customer responsiveness and maintain its competitiveness, by discovering what are the expendable constraints limiting them?

For sure, this set of lessons learned do not apply to matured products or even the typical consulting project. But, staying in the comfort zone in a changing environment is a recipe for failure.

This was originally a talk given at Pixels Camp.