Everybody knows that a doctor makes a diagnosis by taking a careful history, examining the patient thoroughly, doing appropriate investigations and then reaching a conclusion. It is the time-honoured method taught to generations of medical students.

Strangely enough, what everybody knows is wrong — doctors don’t work like that at all, or at least not very often. This is the sort of thing that really happens…

A shy old lady comes in and sits down - I haven’t seen her for a while.

‘I’ve got a little touch of…’ She shakes her head and frowns ‘I can’t remember what it’s called…’

‘Cystitis?’

She smiles with relief. ‘Yes!’ And then she looks startled ‘How did you know?’

‘Just the way you said it.’

Patients would often be startled if they knew how accurately I can guess what they are about to complain of. But notice two things:

First, I don’t really know how I do it. I may think that I do and I may even be right. But I can never be sure that I’m right. What I am sure about is that it’s a very subtle business indeed.

Second, I didn’t tell her she had cystitis, I asked her.

She might easily have said ‘No’. In which case I would have immediately switched to a new and better hypothesis and started testing that. Notice also that I had at that stage only discovered what she thought she had — I had to go on to find out what she meant by cystitis and then find out whether I agreed with her diagnosis.

We do exactly the same thing all the time in everyday life. Here is a very mundane example:

The doorbell rings… My heart sinks.

I am drying my hair having showered after a short morning run with the dog.

It is nearly 8.30am and my wife has gone early to school. Becky’s clarinet sounds from the lounge below. I am about to settle down to the second of the four whole undisturbed mornings that I have arranged for myself during this week’s leave. My mind is trying to pull together the threads of the long-sought overall structure for this book. For once I know exactly where I am going to start. I am very conscious that I really ought to have got going an hour or more ago.

I think, ‘Becky will answer the door — probably it will be the postman with something too big for the letterbox.’

And then I think of a more likely explanation — Mr T has arrived to continue painting the windows (he comes at this time to catch us before we go to work) — ‘My window!!’

My emotional reaction is instantaneous — a sick frustration in my stomach — in far less than a second from having heard the bell. Yesterday Mrs R wanted to clean the room around me, now how am I going to achieve anything if every time I look away from my screen and up to the hills for inspiration I find myself peering into wondering eyes two feet away through my window pane!

The clarinet stops — Footsteps — Silence — I put on my socks. I begin to relax. The silence does not fit with my Mr T hypothesis — Becky should have called me by now. Still silence. An alternative hypothesis gathers strength: Becky’s friend Mandy has called before setting off for the college. But if Becky has for some reason chosen not to call me it would be rude to delay longer before going down.

The only way I can resolve the matter (apart from shouting) is to go down. I finish dressing and do so. On the stairs I hear the faint sound of a laugh from Becky. The type of laugh fits with the Mandy hypothesis. My relaxation is almost complete. I can see Becky’s outline through the obscure glass of the inner door, but who she is talking to? I open the door.

‘Hello Dr Willis, how are you?’

‘I’m very well thank you, Mandy. I thought you might be a postman, a painter, a glazier or a plumber.’

‘Well I’m sorry, I’m not.’

‘That’s perfectly all right.’

One of the contributions that academic general practice has made to medicine has been to point out that doctors work by this process of ‘jumping to conclusions’. But not quite to conclusions; jumping to a series of provisional hypotheses is more like it, because the next and essential step is to test them. The process is rather grandly called hypotheticodeductive reasoning but actually everybody does it all the time in situations they recognize. At first GPs thought it was something to be ashamed of but once it had been given a name it was recognised as a strength.

What I am pointing out is that these provisional conclusions are surprisingly accurate and provide an extremely efficient way of dealing with complex situations. It is another example of the pattern matching that are minds are so efficient at performing. Even in the most mundane moments of ordinary life, our minds latch onto a series of provisional explanations for the things that are happening around us. And even as they do so they go on adapting, strengthening, or replacing these explanations, from micro-second to micro-second. A process of utterly astonishing power and sophistication, when you come to think about it. Or when you try to imagine how you would set about imitating it with a machine!

In the past people assumed that the purpose of the human mind was to establish absolute truth and to guide mankind to Utopia. As a little experiment in thought, let us think what the design requirements would be of a mind whose primary task was quite different from this. A mind whose purpose was to hold the best possible image of reality and constantly improve that image in the light of experience. The building up of an ever more complex and beautiful image of the world, not the focusing down on narrower and narrower absolute truths.

Forgetting the practical details of how the thing would work, such a mind would have to be able to piece together and sustain throughout a lifetime a complex idea, or great hypothesis, to represent the reality out there which was being reported on by the senses. The crucial point is that this great hypothesis could never be perfect or complete, all that the design brief would require would be for it to be constantly improved.

The method by which we would be bound to proceed would be the same as that shown by Karl Popper to be the basis of all scientific enquiry. It would be by ensuring that the great hypothesis is constantly tested by exposing it to as much information as possible about the reality outside.

So first we would need some way of supporting, containing, holding the great hypothesis so that it was preserved and yet free to evolve (a gigantic challenge in itself, but not the present problem!). Next we would need as much and as varied information about the world as we can get hold of. So we would want the best possible sense organs and they would need to be as mobile as possible so that they would report on the widest possible range of reality. That would mean not only arranging means for the sense organs to be transported far and wide (legs, bicycles, Concorde) but also building into the system an inherent motivation for that exploration to be carried out with energy and curiosity.

Next comes the really clever bit. The interface between the great hypothesis and the incoming messages about reality. What are the requirements here?

Simple in principle but unimaginably complex in practice. We want the messages to continually test the great hypothesis; to do this we want to find incongruities. In order to avoid being swamped by the enormous volume of data being collected by the senses, the mind will have to concentrate its energy on a tiny part of the data — the part which conflicts with its existing ideas. It ‘just doesn’t want to know’ about everything else. It wants to get straight on with examining and validating the discrepancies so that it can use them to update its great hypothesis. So, ideally, our design must make the testing process totally automatic and totally invisible.

So what we want is an inconceivably large, totally automatic, totally invisible testing process. That’s all.

Oddly enough, our empirical study of the workings of our minds has already concluded that the thing they are most surprisingly good at doing is selecting for incongruity. And now we have an explanation of why this is so important — of why, in fact, our minds couldn’t work in any other way. The great hypothesis is the same thing as the ocean of congruity — far too large to see all at once — and the incongruities are the beaches around that ocean. And the beaches, although relatively tiny, are the exclusive focus of our attention.

This concept of the way in which we improve our image of the world by constant small modifications is of the utmost importance. It has implications in every aspect of personal life and the life of a civilisation. It applies equally well to the development of the ideas of society whether or not they are expanded to the media scale. It is important because it is so fundamentally different from the way in which we think ideas are developed.

In Chapter 4 I referred to an unfortunate trainee GP being grilled about his reading by a mock examiner.Remember the one who really did make a conscientious attempt to read the journals he thought he was expected to? What is the next question the examiner asks (after ‘Do you do anything else with your time?’) when he has been told that the trainee reads half a dozen journals including the British Medical Journal?

He says, ‘OK, tell me something you read in the BMJ this week…’

The trainee’s heart sinks and his mind goes blank. The examiner thinks he has proved that the trainee was either lying or doesn’t know how to read properly.

What has actually happened is that the examiner has failed completely to understand how people index memories in their minds. He thinks proper reading produces something akin to a photographic record of the BMJ in the reader’s mind which he should be able to recite on demand.

The reality is totally different and immeasurably cleverer than that…

Sure enough, people can teach themselves to memorise information in the exact form it was given to them. Some people do have so-called photographic memories. But this form of learning is far too rigid and far too time-and-energy-consuming to be a practical way of dealing with the experience of a full life (that may be why natural selection has not given us all photographic memories — they can’t be a real advantage). What the sophisticated reader does is to scan through vast quantities of material allowing it to interact with his existing ideas. Everything which is congruous will be ignored, things which are new and incongruous will be selected for attention.

Often, incongruities are ignored anyway because they are simply too disruptive of basic ideas. We can’t constantly be questioning everything. So attention to incongruities is itself selective, and individuals vary very much in their threshold of credulity. The system has been set up with a bias towards stability. This, as we shall see, is another feature of the human mind from which our society would do well to learn.

The mistake the examiner made was to assume that the new ideas which our trainee had selected for retention would be found indexed in the trainee’s mind under the journal he found them in. An elementary mistake, but the conventional wisdom. The ideas are actually neatly filed away, inside the most appropriate memory boxes containing the previously existing ideas that they had served to modify.

So if you say, ‘What did you read in the BMJ this week?’, the mind is blank. But if you hold a short conversation instead, it will constantly pop up memories of relevant things that have been read recently. Each time the reaction is, ‘What a coincidence, I read something about that only yesterday!’ (Notice that the trainee is just as surprised by the ‘coincidence’ as the examiner. The illusion is just as strong for him, that’s why the problem persists.) Once again it is our old friend ‘selection’ at work. We underestimate enormously the extent of our reading because we store away facts in boxes in our minds as we learn them. We cannot open more than a few boxes at a time. But we have been taught to think that the literal, photographic kind of recall is the only kind which is really respectable.

The convergent, literal, photographic approach is a game with the dice loaded in favour of the examiner. It is a game invented and refereed by specialists and it only works in restricted fields of study. While it feeds the ego of the examiner it damages the should-be learner. It kills curiosity and is profoundly anti-educational. It is the most appalling introduction imaginable to a life-time of self-education in general knowledge.

I am certainly not saying that the mock viva was a bad preparation for the real examination. Quite the reverse. The pre-examination course from which the viva was taken as an example was extremely successful in its objective; getting people through their exams. The pass rate was close to 100%. But what the course was doing was to teach a technique of answering a particular kind of question. What we really want to know is whether this kind of educational exercise makes people more or less likely to educate themselves twenty or thirty or forty years on in their careers. That is a question to which nobody knows the answer. All we can do is use our common sense, and there are no experts who can pretend they know better than the rest of us.

My common sense tells me that the doctor must not even try to memorise pages of information. The time for that type of learning was when he was at medical school long before and the basic skeleton of his future knowledge was being laid down. What the mature doctor needs to do (and in fact does do, whether he realises it or not) is to constantly maintain and up-date his mind-image of medicine by exposing it to the whole range of experience and current ideas.

The process is not that of photographing individual facts but the far more sophisticated and useful one of fine-tuning a personal synthesis. The fact that doctors, amongst others, do manage this process into the teeth of the head-wind of contemporary educational effort suggests how very much better they would be able to do it if only contemporary education recognised this fact and exerted its undoubted energy on finding ways of enhancing rather than hindering it!

There are a great many implications once the reality of this model of the learning process is accepted. It provides an explanation for such diverse phenomena as perversity, negativism, criticism, the media emphasis on bad news, the restless search for novelty, and the emphasis on progress and change in human affairs.

It shows us why these things are necessary and inevitable and by enabling us to understand them, it may help us to contain them. And contain them we must in a world which is no longer a limitless jungle in which people fight ruthlessly for supremacy but a closed, finite community in which all must find a place and a purpose.

Although critics often anger people by being destructive and negative they are only doing their job. If it was their primary job to point out good things they would be called something like plaudits. But their job is to pick out incongruities and signal them on the media scale. Their name, rightly, describes their primary function which is to criticise and therefore promote improvement. They are one of many media scale equivalents of the mechanisms which search for incongruity in the human mind.

Critics are just one category of journalist. All journalists really have a common aim; picking out the areas of change and incongruity and highlighting them. A good journalist has a kind of genius for sensing the beginnings of a wave of incongruity building up on the surface of the ocean and then mounting that wave and riding it right up on to the beach so that he arrives just ahead of everybody else in a cloud of spray and glory.

It is just the same with fashion leaders and politicians. The game has all the exhilaration and risk of real surf-riding. It is a very delicate business to get right. Few manage it more than occasionally and even then there is a large element of luck. Don’t forget that media-scale super-selection operates to pick the winners and it is only in retrospect that their success appears to have been inevitable. Many promising waves peter out. Even when you are on a good one, if you move too far ahead and lose contact you are regarded not as a genius but as a crank.