The Psychology of Innovation – we don’t know how wrong we are

  • People , once outside their competence, cannot judge how wrong they are ( and usually think they are right – the illusion of knowledge).
  • People observing groups work, consider groups that change direction (pivot) to solve a challenge as worse performing than groups that set a clear scope and plan and stick to it (when tasks are controlled to have the same outcome and performance regardless of approach).
  • If, given a multiple choice exam, you pick an answer and then have doubts and think another answer may be right – you have a better chance of being right if you change your answer rather than stick with the first answer.
  • People have a strong desire to maintain consistency once committed to a direction and behavior.

These are outcomes from investigations into the psychology of behavior of individuals and groups (from work by David Dunning and Justin Kruger and others). Note the text is my paraphrasing.

When I read these insights from psychology – they opened up a lot of questions in my mind about how we work with Innovation and supporting processes like Agile and Lean startup (MVP/Build-measure-learn).

I have always been asked to think about Agile and Lean startup as models how to work with high risk projects or explore new business models ( i.e. processes for accelerating success in innovation).

They are sold as ways to create small , self-motivated teams, who can move fast and ensure the outcome is quickly adapted and changed so we get close to the real customer value as we develop the offering.

When I started to look at the psychology of behavior, this opened up some other thoughts.  

Lets look at the insights above:

“People , once outside their competence, cannot judge how wrong they are (the illusion of knowledge)” .

When we start activities with high uncertainty and risk – pretty much the definition of innovation related activities – we (the team) will be operating outside our competence by definition.  

From the insight, it also means we will have no idea how wrong we are – but will tend to believe we are better than we are – “Your baby is ugly” is true – and you will not know how ugly it really is.

This means, for Agile and Lean startup processes – the real aim of the process must be to ensure that the team finds out where they have no competence and do something to challenge their knowledge and competence , and then build it up competence to meet the challenge.  

The psychology implies something that is obvious, but not something that we often focus on when starting up Agile / Lean startup – we have to build up our competence. Slightly different to finding the customer pain / gain, fill out the business canvas and so on. 

How many of you have run innovation processes or agile projects where a team is selected because they are a “talent” team of new high fliers. Or been led down the path that what we need is “new blood” with “new mindset” to bring in innovation. 

I have had the privilege of running innovation events and processes with all sorts of mix of teams. There seem to be very little common elements that drive the successful teams, but these two observations seem to hold true consistently (both are needed):

  • Teams that have at least some members with strong relevant competencies in the domain in question do better (eg. compared to pure talent teams with high general competences)
  • Teams that are highly reflective to outside inputs, responsive to make changes and effective in using others, do better (ie are effective at building new competences).

The psychology statement suggests that if an agile / lean startup team do not show they can respond to inputs and develop competences , then we should change people within the team. Also teams that have no basis in the problem (have no strong start point in relevant domain competences) are going to struggle and take a lot longer to get going – usually longer than a large organisation has patience for – and should change their team composition.

We forget , perhaps, a bad team is still a bad team, also in Agile or other innovation processes. Team competences, together with a learning mindset, are a critical part of this. The process does not solve this – only (hopefully) makes it more obvious earlier. 

Having said that – we can also see why the process structure of Agile and Lean startup variants are powerful, from the psychology point of view.

Both demand that the team work in short cycles where they do something, test it works and demo it – then change their next actions based on observations and learning. This inherently forces teams to quickly test their ideas work and then ask external stakeholders to verify it.

External verification and fact based testing on what you have done is about the only way a team will know “how wrong they are” and if they are on track to “get it right” – and so move from the illusion of knowledge to knowledge – provided they change based on learning. 

Essentially, we are , in these processes, building in a countermeasure to a human behavior for the “illusion of knowledge” that works against innovation. This, in turn, suggests why these processes have such a positive impact when applied effectively.

As an aside – we often use the word “customer” to define the group that we need to test / demo with. I think this often leads to confusion and unnecessary debates (“we can’t get the real end customer this time, but Bob will be an OK proxy”). Instead we should use the idea of the most competent “external verifier” we can access, relevant for the question or assumption being explored. 

“People observing groups work, consider groups that change direction (pivot) to solve the challenge, as worse performing than groups that set a clear scope and plan and stick to it”

This one, I think, really suggests why a more formulaic process like Agile works in larger organisations as a framework and why Lean startup has great ideas that can be used to replace more rigid process elements of Agile – and not the other way round.

Stakeholders – or the people who pay for projects – really do not like a process that says we can pivot and change as we go and cannot articulate clearly where we will end. 

We need to be able to change directions and pivot to overcome the bias of “how wrong we are” and yet our paymasters see this as bad performance – instead asking for things like business plans and clear targets within specific time lines.

Lean Startup thinking is the most articulate and clear on need to pivot and learn and how to manage exploration and high uncertainty – addressing the individual and team bias mentioned above – but does not provide any real crutch of reassurance for stakeholders to invest in the process and feel some element of control in their investment. This is problematic, as its always difficult to change peoples mindsets, and if we don’t provide some kind of structure that helps , then its often down to just persuasion, training and case examples of success – slow processes that are difficult to scale.

Agile , however, manages to convey a sense of direction and provides a structure that paymasters can use to help guide their behavior and helps stakeholders feel like the team is driving in a consistent direction with a plan and purpose. It then, through frequent interaction with stakeholders (eg via demos) – allows them to express concerns and be brought along a journey of the team changing directions in a manageable way (from the stakeholders point of view).  

Agile says we have a project vision (clear high level direction) and through its elements like release planning, sprint structure and backlog thinking, allows teams to roughly predict ahead how long things will take without knowing exactly what will happen. If we consider the psychology insight above that observers don’t like teams that change direction, then this is a much more reassuring framework ,psychologically, for stakeholders than one that says we will pivot and change, like lean startup. This suggests that combining the Agile elements that support stakeholders with lean startup elements that are very direct in the change and learn loops could be a strong combination from a psychological viewpoint, to these different key groups.

“If, given a multiple choice exam, you pick an answer and then have doubts and think another answer may be right – you have a better chance of being right if you change your answer rather than stick with the first answer. However, in general, individuals have a strong desire to maintain consistency once committed to a direction and behavior.”

I’ll put the last two statements together. For me, this suggests, as an individual, you have a better chance if you pivot than if you don’t – but our instinct is to stay on the course we have chosen. (Its not quite the conclusion of the psychology trials – but close enough.)

Basically, at the root, we have a mindset that likes to have consistency (and stay within our comfort zone of competence). However, to really deal with innovation and uncertainty, we have to pivot and change based on external inputs and learn as we go.

This means we must train ourselves to manage our bias as individuals not to pivot and change. In this way the principles of both Agile and lean startup are again effective at providing a value set and structure that provides a countermeasure to this bias at an individual level. In particular – both strongly push the bias for fast action and learn rather than planning and analysis.

In summary these statements mean , at the heart of innovation, we have a dilemma of human behavior – we are asked to behave in ways, at the stakeholder, team and individual level, that are not our preference and we are exposed to unconscious biases that work against innovation, like thinking we have it right when we don’t.

Agile and lean startup (and the many derivatives and variants that follow a similar structure) are process frameworks that mitigate the worst elements of human behavior from a psychology point of view and as such are a strong support to innovation – indicating , perhaps, why they can be very effective.

In particular we see there are three key groups of people, who are key in innovation from a behavior viewpoint, that should be addressed by whatever process we decide to use, to help correct for natural bias against innovation. These are:

  1. The individual person, their competence and motivation to engage and learn
  2. The group or team, their motivation to exploit each others competence, engage with each other and external, share goals and jointly learn
  3. The stakeholders or ‘ paymasters’ and their motivation to succeed through the teams results ( typically to earn money / get a return on investment of some kind)

This means that we should be able to merge and modify elements of innovation models & processes freely, provided we retain the key elements that support the preferred behaviors we want in the three groups to support innovation.

An experiment to focus on behavior rather than any specific “process” formula – we did try such an exercise on a technology innovation project.  

We kept the structure of time-boxed sprints and demos – for the stakeholders to be able to follow and feel they can effectively express their opinions and be heard. This addressed the stakeholder need for some consistent structure and address their behavior bias against test and pivot. It also provided the team and individuals with some sense of consistency and overall direction – also a key behavior need that was requested by individuals in the team.

Then we focused on using the rapid assumption / testing loops from lean startup “build – measure -learn”, where teams must ,fact-based, verify top uncertainties rapidly with external inputs and objective testing. We also challenged the team to define multiple possible “design directions” to explore in parallel. This ensured a structure to test how much we don’t know quickly, encouraged the team to exploit each others competence and reach outside for external verification as a foundation in all activities. This addressed the “illusion of knowledge” bias outside the teams competence zone and avoided the trap of following one direction too long (avoid the team and individual preference for consistency of direction). Instead driving quick learning loops. It also reassured stakeholders that progress was being made.

Finally we worked with the team for a day, to practice rapid testing loops, getting verification from external sources and then adapting the next tests based on leanings. This training – where people experienced directly and personally, within a training environment – the benefit of rapid learning loops and following multiple options, closing them down quickly – was seen as the best way to address the individual behavior biases. Feedback reflected this – after getting first learning from very rapid tests – it opened eyes how much can be gained quickly. This re-enforced in the individuals that this approach worked, giving confidence to continue in the same way.

In this way, by thinking about the basic behaviors we wanted to support, rather than any specific process, we merged ideas from several sources to get a simple process that was up and running in a few days. 

Interestingly – we could not impose co-location or dedication into this project. We instead put some simple , effective virtual tools in place with a time each day the team had to be online working on the tasks.  

We also dropped a number of other elements, that are considered key in different processes, but are complex to embed and train (eg strict role definition and splits like Product Owner & scrum master).

So far – it seems to work well – suggesting if we focus specifically on supporting underlying behaviors that drive innovation in the three groupings of people that are key for a project – we can go a long way with simple process structures.

(the views here are my own personal views)

Innovation: a definition

“Innovation occurs when there is external confirmation that a value aspiration gap has been reduced”

This definition of innovation came out of a forum discussion between myself and Don DeLauder, as a response to a question by Eugene Ivanov. I started with the intention of arguing that defining Innovation was a pointless exercise and ended up writing a definition that was thought provoking for me – hence the decision to share it.

First let me unpack the definition a little.

A value aspiration gap: This is a value gap that is opened when a tension emerges between the current state and the aspirations of people and/or entities. This is sometimes referred to as “unmet needs” – but innovation can be wider that just “needs”. If this tension grows it creates an opportunity and if people cannot find a way to close this gap with what is available in the current state, then some form of innovation is needed to fill the gap. In other words the need for innovation comes when the status quo fails to generate results that meet our aspirations, whether now or in some predicted future. In that case we need to create more value-we need to innovate.

External confirmation: This is when someone or something external to the creator / inventor confirms that a value aspiration gap has been reduced (ie some aspiration has been all or in part met) by assigning recognizable value to the innovation.

This idea that innovation occurs when we have confirmation external to the creator / inventor is broad – so here is an attempt to place some “boundary markers” . The idea being if one marker is crossed – it may be innovation – but as more are crossed , the certainty that we are looking at innovation goes up:

a) Confirmation of Financial value has been assigned to the innovation by a user – this could be anything from a third party willingness to donate in kind, to more traditional payment

b) Availability (the creation is available / accessible to others)

c) Shown to have addressed a real value gap – ie other people are using it to do something they did not do before.

d) Recognised by independent sources as creating value (eg technical magazines, reviews, trade fairs and the like)

So lets take this definition with the above clarification and run some thought experiments to test the definition in action.

Innovation covers a lot of ground often expressed in terms of “incremental” or “disruptive” – and everything inbetween. A definition needs to be something others can apply and use as a filter to say what is , and what is not , innovation. For our thought experiment – we need to try and identify things that could be innovation, then see if the definition helps show if they are inside or outside the boundaries of what is innovation, as set up by the definition.

To do this , three examples are picked for the thought experiment (coming from the discussion mentioned above).

At one end we can take the birth of quantum theory – here I am thinking of Einsteins theory that won him the nobel prize. In the middle , lets place Betamax – the innovation that lost to VHS (before both were overtaken by DVD , BlueRay and now streaming). At the other end we place any invention that failed.

If we take quantum theory – clearly this has been the foundation that has led to many breakthrough technologies like Semiconductors, in turn leading to things like strained quantum well lasers and then to CD players and the like. However – this fails to show a value aspiration gap. It takes time from theory breakthrough, to people , to understanding and interpreting , to new technologies , and during this , people start to develop aspirations for things that were impossible to imagine at the time of the breakthrough , creating value aspiration gaps – and then innovations start to occur to close these gaps. So in itself the theory is not innovation, but clearly the birthplace of much of our modern world. Innovation occurs later.

The failed invention loses out because , although the inventor has percieved some value aspiration gap – there fails to be any external confirmation that this is a real gap – so it is a “wannabe” innovation – but by the definition is not innovation.

Finally Betamax. This cleary fits the definition. People bought it , it was recognised as innovative, it was used and so on – all the indicators of external confirmation being met. Betamax failed due to normal business dynamics – in my simple version, both closed a value aspiration gap, but VHS got a better handle on the real value aspiration gap (wide content selection with good enough technology) so won the business battle over time.

The outcome of the thought experiment “feels right” – ie it successfully and clearly separates what is and what is not innovation in a way that, although utilitarian, seems to work and give meaningful answers. So its a good working definition. The boundary markers can, I am sure, be improved, but they seem also to work.

One interesting thing about this definition is that there are no words in the definition that states an innovation must be something “new” or “created”. These are two words often used when trying to define innovation, but make a sharp definition very difficult.

As long as the value aspiration gap is reduced, we have innovation. Creation is implied but not needed as part of the definition. Newness also is implied as an outcome, but not needed. It becomes instead a property people can assign as an outcome – things are percieved as new if they close a value aspiration gap – but perhaps are not new in some other context.

I hope you find this definition also of some use. Comments always welcome.

If people want to read the original forum discussion:https://www.linkedin.com/groups/4435998/4435998-5885738242841546754

Apple : yet another analysis of Apple’s success

I wrote this a while ago (2016 I think, maybe earlier) – and its being sitting in draft doing nothing. I thought it time to put it out there. Have added a few comments to make it seem like it was written recently…enjoy

I have been subjected, like many of you I suspect, to Apple being used as the pinnacle of innovation in many a business conversation – usually with someone (internal and external to the company) trying to convince me that what we do today is inadequate and if we just did “x” , then we would be much more innovative , profitable and experience exponential growth “like Apple” (used to imply we are hopelessly behind, and its obvious what Apple is doing that we are not – but with little further detail). This has thankfully died down since Cook took over – but still is brought out now and again.

Like many people , I have been an Apple watcher over the years, and formed my view of Apple between 2001 to 2010 (iPod launch to iPad launch in “Apple Time”) – and it has not really changed since then. I am sure this point of view has been expressed – but I actually have never heard it – and yet another Apple talk by someone recently got me thinking I might as well add my version to the 1000’s already out there.

Apple to me is a company that has truly mastered Technology Management, Product management, Product development, Supply chain integration and above all launch disciplines , combined with a unique attention to detail in customer experience. All being traditional disciplines and not much to do with the type of innovation, implied in the introduction, people sell me about Apple.

My first realisation of this was when I bought the first iTouch – for those of you who have forgotten – it was released as a touch screen version of the iPod – with no apps store and no bluetooth. I then listened to Steve Jobs, in one of his live streamed events (at that time you needed to download the event), announce the Apps Store , and that everyone who had an iTouch could upgrade to the Apps store for an amazing discount rate of $25 (or something like this). I happily paid like others and was amazed at how little backlash there was to this charge. A little later he did the same for the launch of the next OS X and charged something like $50 to those that had to upgrade. Again little backlash.

I started to pay more attention to the product releases coming out of Apple. The lack of bluetooth at a time was completely against the de-facto rules of launch , as it was standard with competitors, yet still the iTouch was smash hit and with very little backlash from consumers (or non really I could determine at the time, except by reviewers). Then the perfect positioning of the iPhone on stage as a 3-in-1 device (phone, internet, music/messaging — the third always got a bit grey). Then the positioning of the iPad as the missing piece in our lives with iPod/itouch/iPhone at one end of the mobile spectrum and the Apple Air / Mac at the other and the blatantly obvious (when Steve showed us) gaping market hole between the two just waiting for the iPad / mini iPad to fill.

Each of these events were announced in advance on stage – with prices! I still consider that series of live stream events a better education in product strategy, product positioning, launch strategy and pricing strategy than anything in a text book. If you can find it – just look at the stage performance where Steve Jobs launches the iPad. The perfect start by redefining the product category of mobile devices – the perfect identification of a missing product position within that category (the iPad) and then perfect delivery into that gap. Why bother with text books or executive MBA – as an educational video on how to do product strategy, product positioning, value propositions and product pricing – I have never seen better.

Looking a little beyond these headline events – what does this show us? Well – to launch the itouch , then enable an upgrade to apps and next gen. OS X shows us that this was a product platform with these steps already designed in at the start. The same with iPhones and the upgrades that followed. To do this is not the sign of a company developing an MVP (Minimum variable Product – the textbook expectation of highly innovative companies developing business models and pivoting). Rather it is a development process and product that has been carefully thought through not just for first launch , but also for subsequent product releases.

This thought was re-inforced when meeting a US californian based consultant with insight to product development at Apple. He commented that at an early gate (note gate – so traditional set up) , the project team had to not only outline the product concept , but also the next (2 – 6?) releases that would come from that product family. In addition to this, there would often be parallel teams working on the same product or product sub-system – ensuring competition to drive innovation. Great for those companies with the money to do this.

Then there are the launches. The stage performances are used to announce world wide launches – but typically with US or a limited set of countries getting first release. Often other countries not even given a date at first. As the launch started , other dates would be announced for other countries. In terms of supply chain , I always thought this was a great way to manage the demand curve – so consumers are not disappointed and have a sense of the product being released all at once, but the supply chain using country roll out dates to control the demand curve , so it had a lever to pull to enable factory output to follow demand without significant customer backlash against product delays – Again that ability to manage customers feelings towards the company. Although even this has challenges if the demand is explosive against prediction, like the iPhoneX.

Then there is the complexity management. The apple product range is a supply chain wet dream with tight control of variants, only minimum variants available and a standard , incredibly disciplined and effective phase out (launch a new version – keep only the low end version of the old version – discontinue the others immediately to ensure a hard change over). Ensuring amazing scale with few variants. I don’t know how many SKU’s there are of the iPhone, but it can’t be much more than 100 – a number most companies would die for , in a product group. It also effectively allows new platforms of a product to be launched with low re-use from old platforms (so large room for manoeuvre to introduce completely new technologies, processes etc) but maintain supply chain scale and simplicity (and thereby cost that others can only achieve with partial platform upgrades, stretching lifetime of modules to keep costs low).

Finally the technology mastery. In each area – there is no doubt Apple has identified key technology leadership targets and invests heavily in them. Unlike many companies , it clearly believes process technology leadership is as important as product technology – with each product launch seeming to bring technology jumps in production areas it believes enhances its customer differentiators (like the single block Alu bodies, to the fused screens and so on). This selection of key technologies to lead is clearly also seen in the product technologies applied. And of course – they build this into the promotion – name another Company that will split its technology and show the insides in promotion videos? Not so many on my list – except maybe high end Swiss Watch makers.

All of this , for me, implies a company with an incredible discipline in technology management, product strategy , product development , go to market and supply chain excellence – not a company pushing innovation fast , at the limits , with a series on MVPs and rapid iterations, exploring uncertainties and business models. It implies a company that takes the integrated product development disciplines to the limit – fantastic product platform thinking infusing all steps, amazing product strategy built on customer insight – out of this world technology roadmap thinking tied to product strategy and customer insight – far thinking product development discipline with competition built in and supply chain and market launch completely in sync.

The magic for me is of course the same as all others point out – spotting new “mega” categories from the customer insight (however this is arrived at). This implies a user oriented culture deeply embedded in top management (or I guess most people would say in Steve Jobs personally – but not so sure), as well as in the organisation.

But from this point in – it is just out of the world execution and top management “ice in the stomach” (to use a Danish expression) to follow through convictions. I am sure Apple did its fact finding and saw that customers at that year (launch of the iTouch) – were not so bothered about bluetooth. They , I am sure, like all others , saw it was coming fast, and others would and did include it. But they also knew that it was not yet a key differentiator – so were willing to cut it from the development of the early versions – saving time , cost and speeding up development I am sure – knowing they would add it to the platform next year. To keep this discipline , plan to release the updates and do it, and then repeat it consistently every year- coordinating market, product development , supply chain and sales so seemlessy (from the outside) – using systems like country release delays to manage uncertainty – is truly impressive.

Similarly for the apps store addition to the iTouch. I am sure they were in the middle of apps store development , but could see they had enough differentiators to launch an iTouch without the store, with huge success. So instead of waiting , did that – cashed in – always making sure to ask for payment where they can see they have a unique differentiation to offer. Then cash in when they have the app store (another extra payment for those with older iTouchs) and cash in on upgrades and again in on OS X. They don’t charge when they can see the differentiation is not sufficient to ensure customers are more than happy to pay – so their pricing discipline again seems to be a refined science routed in customer insight.

All of this for me indicated a product Development, marketing and supply chain excellence company that is world class. It does not come over at all as an innovation company with Google like “20% – do what you want” innovation processes or anything like that. In fact all the elements seem very traditional management disciplines in one sense – but taken to a competence level rarely seen. 

Then of course Apple adds one more layer. The almost obsessive attention to detail of customer experience at every level is close to unique. All tech Companies clearly now focus on user design, but where as I always feel places like Google do it case by case, for each product – then stitch it together – Apple seems to be able to embed the ecosystem experience deeply into the individual product experience. Clearly with advantages and disadvantages – I would not be surprised that this overall experience thinking means they did not see as clearly as say Amazon, the one trick Echo dot like activating music with voice – even with Siri being around forever.

I always remember the complete shock at the apple map launch failure. In itself it was not really that much of an issue – but to see apple slip in what seemed to have been over 10 years of launches with complete 100% attention to customer experience that never missed the mark – then have them make such a customer experience error – seemed just wrong. There have been other launch failures – but not with such a clear miss on customer experience.

So innovative or business excellence at its best? One thing is “there are no shortcuts on the road to success” seems to be true of Apple – there seems a hell of a lot of hard work , discipline and process excellence going on behind its “innovation success”. Maybe like the saying “chances seem to come more often to those that work hard” , we could say that “innovation seems to come more often to those that drive for customer experience and business execution excellence”

These views are my own and not related in any way to the company I work for.