So, you think you understand the geology and your data is all ready to go. You’ve spent days, weeks, months drilling, sampling, logging and interpreting. The CEO is breathing down your boss’s neck demanding an immediate update so it can be released to the market. After all… those shareholders are waiting to see exactly what their money has bought! The pressure is on. It’s time to start estimating. Ready to turn on the old computer and feed data and parameters in to generate an estimate?

Data + Parameters = Estimate…. What

Despite all the pressure and the demands for ‘real’ visceral outcomes, it pays to slow down and do the hardest part of any resource geologist’s job… Thinking.

In many ways, we need to treat our modelling efforts like safety management. Every single on-the-job safety program I’ve encountered over the years has had a similar thread. The first couple of steps are to stop and think. Avoid rushing in and take that extra breath. Observe the environment, consider the task, look at the tools at hand and assess the risk. We focus on ensuring people avoid injury (or worse). We focus on ensuring people go home in the same state of health as they had when starting work. And for good reason – no one should have to face the prospect to being injured just to earn a quid.

I like to apply the same concepts to my models. The better my thinking the better the model. I need to stop, assess the situation and make appropriate decisions. Only then is it worth tackling the estimate itself.

You see… sometimes our thinking can be fatally flawed. While we like to think of ourselves as rational beings the truth is we are more rationalising than rational. We can convince ourselves just about anything is ‘true’ if we put our minds to it. Let me give some examples…

Crisis calls

I recall a rocky start to the consultant-client relationship on one occasion. This particular client was in the process of recommissioning an open pit operation after a long and arduous feasibility study. They were having some serious problems. Despite every effort, the expected grade (from the resource model) simply wasn’t presenting at the ore treatment plant, or in grade control for that matter. I was engaged by the COO as part of a larger cross-discipline team to investigate the root cause of their grade shortfall and identify corrective actions.

After the investigation (which was fairly simple) I walked into a meeting with the CEO, the site General Manager, the Chief Geologist and the rest of the investigation team. The CEO kicked off the meeting by saying, “Right… we know there’s nothing wrong with the resource model… what is going on?”

He’d fallen into one of the biggest cognitive traps around. A form of ‘group think’. The company had spent millions on resource drilling, sampling, data validation, consultants. They had engaged ‘industry experts’ to help with the modelling and estimation. They had gone well beyond what would be considered industry standard practice in terms of data collection and validation. How could the model possibly be wrong? He had a competent Chief Geologist and he had faith in the rest of the geology team. They had done an excellent job – it was all documented and logically presented. There couldn’t possibly be a problem with the geology or resource estimation. Or so he thought.

This case of group think was one of the worst I have ever come across. It ruined several careers and very nearly ruined the organisation. And, it was all because of a combination of culture and belief. The Chief Geologist, the Feasibility Study Manager and the Chief Project Development Officer were all very opinionated and strong personalities. They formed a solid front facing off against the rest of the organisation. They were the future. They were cleverer, more intelligent and more business savvy that everyone else. They deserved success. No, they simply were success. In the face of that sort of aggressive stance anyone who criticised their work was quickly shut down (and often removed!) This attitude even impacted the CEO. The message he received was “We are the best – you employed us to do a job and we are getting it done. We are the experts and you can (must?) rely on us.”

Now, I’m all for self-belief and self-confidence. You can’t be successful if you don’t have faith in your abilities. But… and it’s a big but… being overly confident can be more dangerous. Particularly if it leads to dismissively ignoring the opinion of others, or selectively hearing only ideas and opinions that reinforce your own.

This is exactly what happened for this client.

Remember this was a project that was re-starting an old operation, changing what was a selective underground, high-grade mine into a bulk, low grade open pit. During the days of underground mining the operation had a reputation for consistently over producing. The ore treatment plant produced much more metal than was estimated in the resource model.

You might think that under estimating is a good thing. I’d argue it’s as bad as over estimating. It leads to bad practice and reliance on ‘doing better’. It can lead to sterilising what should be economic mineralisation. It can lead to bad operational and financial design, over (or under) capitalisation.

This history of under estimating and over producing ended up pervading the entire geological effort at the site. Every aspect from drilling to estimation was aimed at proving the same effect would occur in the new open pit. Everyone expected their resource modelling and estimate would be biased low compared to the ‘real’ production. The geology team went looking for evidence to support this hypothesis. And… they found that evidence.

They found:

  • The closer the drill hole spacing the higher the grade of the mineralisation;
  • There was a bias between the reverse circulation (RC) samples and the diamond drill hole samples. The diamond drill holes were always lower grade than the RC;
  • In several test pits excavated during the feasibility study the grade of the bulk samples was higher than the expected (modelled) grade.

It was a pretty conclusive story. The grade during mining and ore treatment was going to be higher than expected.

Unfortunately, the conclusions drawn from these observations were entirely incorrect. In incomplete understanding of sampling and statistics coupled with a self-serving bias lead to fatal flaws. There was an alternative and more correct explanation which was totally ignored. Contact me for more info.

Consequently, the grades in the resource model were factored up (increased) after estimation to reflect the evidence collected by the geology team. In some cases, the factored grade was 40% higher than the estimated grade.

And everyone from the CEO to the project geologists believed this was the right thing to do. After all… the old underground mine always produced more metal than predicted didn’t it?

As I said, group think at its worst, but there was more going on. Several other cognitive biases and cultural issues in the mix. There was emotional investment, issues of salience, noble cause corruption, and fundamentally flawed logic. There was selection bias and self-serving bias.

Here are some symptoms that made this obvious to an impartial observer:

  1. No one questioned the application of very large factors, despite factoring being inconsistent with industry norms. Arguments for factoring were based on poor understanding of statistics and experimental design;
  2. The operation was considered ‘special’ and ‘unique’. Different to every other mine in the world and therefore it needed a unique approach to sampling, interpretation and estimation;
  3. There was clear evidence of ‘consultant shopping’. Many consultants had been involved with the project at one stage but their involvement tended to be short term with a low level of repeat business. To me that points to either the consultant not wanting to be involved (reputation risk) or the client not agreeing with the consultants’ recommendations;
  4. Consultant reports were being selectively quoted. Some very high profile consultants were engaged and wrote quite critical reports. As many consultant do however, they also highlighted areas of good practice. These positive comments were selectively quoted in the feasibility study while the critical points were not included;
  5. Evidence that didn’t fit the preconceived idea of under-estimated grade was excluded or written-off as poorly executed. If the data didn’t fit the belief it was ignored;
  6. The operational production target (in metal produced) was fixed and announced to the market well before the feasibility study was complete. That locked in the answer and the organisation simply couldn’t back down from its messaging;
  7. The risk assessment (yes there was one) failed to evaluate the consequences of factoring the resource estimate. At no stage was the unfactored estimate used for an alternative evaluation. Incidentally, if the unfactored estimate had been evaluated the operational design selected in the feasibility study would not have passed the organisation’s investment criteria.

Possibly the worst aspect of this entire story was the lack of any real desire to examine any alternative causes for the observed data. No one tried to think outside of the box. Sadly, the group think and self-serving bias prevented any attempts to really understand what the data was showing. It was a case of success at all costs – the preconceived operational design and production target drove the study process. Anyone with a dissenting perspective was quickly shut down.

Despite the flawed thinking the organisation and the operation survived. The investment never performed to expectation and there was an extreme level of management attention required to placate shareholders and stakeholders alike. It was pure luck that the metal price exceeded feasibility study predictions… And yet a few simple steps could have prevented such a poor outcome and arguably resulted in a much more profitable operation and certainly one that was more enjoyable to work at! For any preventative measure to work however, there must first be recognition of need.

It’s that safety mantra – Stop and Think. A powerful message and one that should be applied to resource modelling and estimation too.

More Failures

OK, that’s a true but extreme example. Lest you think this sort of thing couldn’t be common or couldn’t happen to you, here are some other examples:

  • Excalibur Mining Corporation
    • 2009 resource estimate for Tennant Creek projects 3.6Mt @ 10.16g/t Au (1.17Moz). Including 1.1Mt @ 18.0g/t Au Indicated Resource
    • 2010 resource estimate for Tennant Creek projects 2.1Mt @ 4.1 g/t Au (0.28Moz). All Inferred Resource
    • Downgrade attributed to errors in database including mine workings used for depletion
  • BMA Gold Limited
    • Twin Hills ‘Area 2’ resource downgraded from 277Kt @ 21.6 g/t Au (193Koz) to 214Kt @ 10.9 g/t Au (75Koz)
    • Downgrade attributed to “significant reinterpretation of the geology”
    • Triggered 30% plunge in share price, early mine closure and appointment of Administrator
  • Deep Yellow Limited
    • 37% increase in resources announced January 2004 (Mikado operations)
    • April 2004 Deep Yellow reports grade control indicating reserve model underestimating contained gold by 10%
    • May 2004 Deep Yellow reports recovered gold down 18% on expectations due to “ore continuity issues”. Mining suspended.
Fast-slow thinking
Interested in how thought process affect your life – this is worth a read.

The list goes on and it’s not limited to juniors or explorers. Some of the largest mining companies in the world have had similar issues. The BHP and Billiton merger in 2001 arose from a string of poor investments by BHP in the last 1990’s including Hartley Platinum (below expected head grade due to ground conditions and dilution), Beenup mineral sands (high clay content and abrasiveness). Rio Tinto inherited problems at the Mineracao Cabacal operation in the 1990’s (>70% decrease in reserves).

 

 

That pressure to complete a model and make an estimate can be overwhelming. Stopping and thinking is sometimes the hardest thing in the world. However, it is one of the most valuable ways to avoid costly mistakes.

“The moment one has offered an original explanation for a phenomenon which seems satisfactory, that moment affection for his intellectual child springs into existence; and as the explanation grows into a definite theory, his parental affections cluster about his offspring and it grows more dear to him. While he persuades himself that he holds it still as tentative, it is none the less lovingly tentative and not impartially and intemperately tentative”  TC Chamberlain (1897).

Prevention is Better than Cure

Hopefully I’ve convinced you of the value of stopping and thinking. Great, but exactly what do you think about? There’s not much point in taking the time and energy to prepare for your modelling and estimation unless you have some understanding of what you should be considering. This thinking space is one reason the JORC Code requires five years of relevant experience in the style of mineralisation or type of deposit and in the activity, being undertaken. Five years actively working with the types of problems and challenges presented by the deposit you are modelling – and in modelling itself. That five years isn’t just about surviving, it’s about learning, seeing, experiencing. Getting to know the levers and drivers of resource quality and economics. I mean, there’s nothing quiet as educational as working in an operating environment, being responsible for delivering outcomes and living through the inevitable differences between your predictions and reality.

Each style of mineralisation has its own unique modelling challenges. Each estimation approach has its own foibles. With that in mind however, there are common themes. Common aspects you should stop and think about before you push the button on that old PC. Here are just a few – you never know they might just help you avoid group think and self-serving bias and prevent yet another resource-related project failure.

Idea #1.

Remember – it’s a model. As George Box famously said, “All models are wrong, but some are useful.” The one thing you can be certain of is that your model will be wrong. It’s a question of how wrong is it, and in what way is it going to be wrong? Your first stop-think point is to understand what your model is going to be used for and who is going to use it? I think intimate knowledge of where your model ‘fits’ is essential to good modelling. Take, for example, the difference between Rutherford-Bohr atomic model and Erwin Schrödinger’s quantum mechanical atomic model. Neither model is ‘correct’ as we currently understand but they each have useful predictive properties. It’s the same with your resource model. You need to understand what you are trying to predict and why. Understand the critical properties – what matters and what is less important. This includes global vs. local precision and accuracy, the importance of extrema and distributions vs. averages and the spatial resolution required.

Idea #2.

Go back (again) to the geology and the geological data. Check if it is fit-for-purpose for the type of model you need. What are the strengths and weaknesses inherent in the data? Double check – have you excluded any data because they don’t fit your preconceived ideas? Have you excluded or ignored observations that seem inconsistent with the other data? If so, stop-think. Make sure that data really is worthy of exclusion – avoid a selection bias. Just because you don’t understand what the data is telling you doesn’t mean the data is wrong. Maybe you need to change your mental model. Watch out for those ‘everyone knows’ moments. After all, once upon a time everyone knew there was no such thing as a black swan.

Idea #3.

Stop-think about what the geology is telling you. Ok, you are probably working with a genetic model – maybe intrusion-related gold or a porphyry copper system and you need to be a bit wary of pigeon-holing your thinking based on what the genetic model claims, but… If you are going to work with a genetic model then use it to its fullest. What does the model predict in terms of metal distribution? What controls grade, texture? What elements are associated and what are contraindicated? What should the alteration and geochemistry look like? What about the structure? Do you understand the fluid paths and traps? Do you understand the geometric drivers? Are there useful fractal patterns, self-similar at different scales?

Idea #4.

Stop-think about that the statistics are telling you. Yes, the statistics. Even without going into full modelling and estimation mode you should look at some basic stats and try and understand what they mean in terms of your geological model, your domain model and your estimate itself. In the example I discussed previously, not understanding what the statistics were signalling was one of the fatal flaws that lead to those huge and unjustified factors. Hint, 90% of the time you will be looking at skewed distributions, not a normal (Gaussian) distribution. That matters. Skewed distributions are not as intuitive as normal distributions (as if any statistics were intuitive to most of us!) Examine the comparative statistics of different sample types, different drill campaigns, different geological units. Look for clustering and/or under-sampling. Look for areas that are more or less variable. Think about ‘support’ (and if you don’t know what that means talk to someone who does).

Idea #5.

Talk to your customers. Talk to the engineers, operators, managers, metallurgists. Find out what’s important to them and equally what they really don’t care about. I’ve lost track of the number of models I’ve seen where the first thing the planning engineer does is strip out 50% of the fields and make up some of their own fields to suit their evaluation system. I’ve even had one case where the site engineer disbelieved the geology team’s resource model so they did their own (independent) estimate! If you understand what’s important to your customer you are much better placed to be able to provide a useful product. That’s at least half the battle won!

Idea #6.

Think about scale and resolution. Think about ‘selective mining units’. Think about the types of decisions that will or will not be made on a block-by-block basis. Do you really need very small (sub) blocks or will larger blocks/panels suffice? What is the likely difference (if any) between block size and selection size. This can be a complex question, more so in underground operations.

Idea #7.

I’ve said it before but it deserves its own stop-think action. Understand if your model should represent averages or distributions (variability). Understand the impact of any smoothing that goes with al weighted average estimation techniques (yes, that includes kriging). Do you need to understand how variability may impact on downstream processes? In my opinion the answer to that one is almost always yes. Do you need to understand the with-in-block grade/tonnage distribution?

Idea #8.

Stop-think about the likely project economics – before you start modelling the geology and grade. What is the likely economic cut-off grade? How does that relate to any potential geology/grade domains you may model? Is the cut-off close to the average of the deposit? Is the cut-off close to any grade domain boundary you plan on modelling? These factors will impact on the quality and usefulness of the model.

Idea #9.

Look at outlier values. Are they really outliers or do you simply not understand what the data is telling you? Before you start adopting some top-cut, grade-cap or spatial grade restriction stop-think. Are these apparent outliers simply due to data density? If you had more samples what would these outliers look like? Go back to the geology and the logging are the outlier samples the same style of mineralisation or something different? If they look different should they be in their own separate domain – even if you can’t define it spatially?

And last… Idea #10.

The most important idea of them all. Stop and think about what could be different. Assess the degrees of freedom in your knowledge and data. Develop alternative views of the world and consider the other 9 items in this list using those alternative views – is there one that looks better or as good as your original thinking? All too often we jump to a solution and fail to investigate what could be. Consider:

  • What would the data tell you if every hole was 10% longer?
  • What would the data tell you if there was twice as much?
  • What would the data tell you if you eliminated 50% of the holes?
  • Have you adopted the parsimony principle – the simplest explanation that fits all the data? The explanation that requires the fewest deviations from the available observations, with the least irrelevant detail?

I dare say if my client with the factoring problem had stopped and thought through these 10 items they wouldn’t have ended up in the mess they did. They managed to avoid being another tragic company-killing resource estimation failure by the narrowest of margins. It’s incumbent on all of us as resource practitioners to learn from the lessons of the past and improve.

 

The definition of madness is doing the same thing over and over again and expecting a different outcome.

 

 

 

 

2 thoughts on “The truth about estimation #5 – Nothing is wrong with my model!

  1. Thank you for this article, when reading it feel like you are beyond my back and preventing me what I won’t to do in such case. This is very interested for all resources geologist or for geos you dealing with data (modelling, production etc)

Leave a Reply

Your email address will not be published.