Flawed Risk Analysis

Are you prepared to pay for a major TANKER spill on BC’s coast?

During our participation in the Joint Review Panel hearing process, Concerned Professional Engineers and many other groups put forward a large number of concerns and challenges to Northern Gateway’s proposal.  The Joint Review Panel approved this project in December 2013 without providing any meaningful answers to our concerns.

 

Northern Gateway’s Risk Calculations are Flawed and Unscientific

[hr]

What is the Quantitative Analysis?

View PDF Whitepaper #1 – Flawed analysis, irresponsible approval

What is the Quantitative Risk Analysis?

In order to calculate a risk of tanker spills for the BC coast, Northern Gateway hired a Norwegian consulting company, Det Norske Veritas (DNV), to complete a Quantitative Risk Analysis (QRA) of the tanker shipping aspects of the project.  The QRA is an attempt to estimate the risks of a tanker spill along the proposed shipping route for the lifetime of the project.

View PDF Quantitative Risk Analysis

To perform this calculation, DNV relied on a private database of historical worldwide tanker shipping incidents called the Llyod’s Register Fairplay (LRFP).  Put simply, DNV added up all of the tanker incidents (groundings, founderings, fires, explosions, collisions, etc) that have occurred all around the world and divided them by the total number of miles traveled.  This produces an “incident frequency” – basically a measure of the number of incidents per nautical mile of tanker travel.  This data is then “scaled” in order to adjust it to the specific features of the BC coast: the route geometry, the weather, the tanker traffic, etc.

In general, CPE agrees with this approach as it uses real-world historical data, rather than simply relying on theoretical models.  Unfortunately, however, there are a number of serious flaws in the details of this analysis.  As with many large and complex engineering projects, details can have a huge impact on the safety of the undertaking.

Why the Quantitative Risk Analysis is flawed

Summary

  • The starting database of incidents is known to contain errors
  • The starting database of incidents is private, and cannot be peer-reviewed
  • The database was filtered in a way that may reduce the predicted risk of a spill on the BC coast
  • Risk scaling factors are chosen without justification and no records are available for examination
  • The risk calculations do not account for the increased traffic from LNG developments
  • The effectiveness of mitigating factors like tugboats is not proven

The database is known to contain errors, as Enbridge’s consultants acknowledge

View PDF March 18, 2013 JRP hearings

During the March 18, 2013 Joint Review Panel hearing session in Prince Rupert, a lengthy discussion took place regarding the known issue of under-reporting of incidents (fire, collisions, power loss, grounding, etc.) in the database used by DNV (March 18, 2013, line 31302). Although DNV acknowledged that the database significantly under-reports incidents (sometimes by more than 90%), they stated that these shortcomings are made up for by the fact that the number of reported incidents is used to calculate both the incident frequencies and the conditional probability of an incident resulting in a spill of oil. In effect, they say, any under-reporting of incidents is cancelled out.

While this may be true on paper, it could be far from true in practice. DNV’s analysis assumes that the conditional probability of incidents turning into spilled oil is the same in Douglas Channel as it is in the rest of the world, on average. We challenge this assumption and propose that the conditional probability of an incident turning into a spill is likely to depend on the width and geometry of the channel in which the tanker is traveling. In other words, if a tanker momentarily loses power in an open channel it is unlikely to run aground and spill oil, whereas in a narrow channel or a channel with tights corners (Douglas Channel), it might be much more likely to run aground.

The database is private so the data and calculations cannot be checked

The database of tanker statistics upon which the QRA is based is privately owned and access costs an estimated $50,000.  Multiple groups requested access to this database in order to confirm the numbers presented by Enbridge and DNV, but requests for free access were denied.

Without access to the underlying information, it is impossible for an independent party to confirm the analysis performed by DNV, a paid consultant working for Northern Gateway.  We are unable to confirm whether the database is accurate, whether DNV’s filtering of the data is appropriate, or whether the incident frequencies they derive are correct.

In effect, we are left to trust the work of a consultant paid by Northern Gateway, without independent confirmation.

The database was filtered incorrectly, in a way that reduces risks

DNV, Northern Gateway’s consultant, filtered the database of worldwide tanker incidents before they derived their numbers.  They only used data from 1990 to 2006 as they claimed that this range more accurately represents the fact that many improvements have been made to tanker design and operation over the past several decades.  For example, the use of double-hulled tankers, tug escorts and improved navigational equipment.

Although some filtering of the data is justifiable, we believe that DNV has excluded at least two major spill incidents that could impact the overall spill risk calculations.

MV Braer

The MV Braer ran aground off the coast of Shetland, Scotland in 1993 after a broken pipe allowed seawater to flood the engine and the ship lost power.  No rescue operation was attempted and the ship split into three parts, spilling the entire cargo of nearly 85,000 tonnes of oil into the sea.  DNV excluded this incident because the Braer was a single-hulled ship, whereas modern oil tankers are double-hulled.

The problem is that there is no evidence to prove that the Braer incident would not have resulted in a spill had it been a double-hulled ship.  Just because a ship is double-hulled does not mean that it is unsinkable when pushed against the rocks in heavy seas.

Exxon Valdez

In  1989 the Exxon Valdez struck a reef in Prince William sound, Alaska, while carrying 172,000 tonnes of crude oil.  Although the ship remained intact, the hull was punctured and an estimated 35,000 tonnes of oil were spilled into the sea.  The Valdez spill was not included in the database because it happened in 1989, one year before the cutoff for DNV’s filtering of the data.  Also, the Valdez was a single-hulled vessel, so even if it had happened a year later, DNV would have excluded it.  Again, we take issue with this filtering without convincing evidence that a double-hull would have prevented this incident.

While the exclusion of a small number of large spills does not sound like a big problem, tanker incidents are very rare, so leaving out a major spill like the Braer or the Valdez can have a significant impact on the data.   Without access to the database it is not possible to determine the impact of these omissions on the overall probability of a spill on the BC coast.

Do you trust the data filtering performed by a Northern Gateway consultant?

Risk scaling factors were chosen without justification

In order to apply the historical worldwide data of tanker incidents to the BC coast, scaling factors were used to account for local conditions like route geometry, weather, traffic, and currents.  If a certain characteristic of the route is considered riskier than the world average, the risk factor is greater than 1, if it is less risky, the risk factor is less than 1.

These scaling factors have a critical impact on the calculated risk of a major tanker spill.  As the only document that comprehensively deals with the risks of a tanker spill, we presume that the Joint Review Panel relied on these numbers when they determined that a major spill was unlikely and they approved the project.

Unfortunately, there are many, many flaws with the choice of these scaling factors.

Do you trust their numbers?

The QRA assumes tankers spend 10% of their time in waterways like Douglas Channel

The most common tanker incident is grounding – where the ship runs aground.  Naturally, a tanker can only run aground when it is traveling near land.  The worldwide incident frequency number that DNV calculated must be scaled to account for this fact.

As a simple analogy, imagine an engineering company wants to build a new pipeline across British Columbia but citizens are worried that avalanches could rupture the pipeline and spill oil into the environment.  To calculate the risk, the engineering company adds up all of the incidents in which an avalanche has caused a leak from a pipeline and then divides this number by the total length of all pipelines.  They could then say they have calculated an incident frequency per km of pipeline  for avalanche-induced spills.  This number could then be used to estimate the probability of an avalanche and a spill based on the new pipeline’s length.

An astute observer would notice that only a very small portion of total pipelines are built in areas where avalanches occur and all of the pipelines that are built in the prairies should not be included in the calculation.  If only 10% of pipelines are built in mountainous areas, the other 90% should be excluded and the incident frequency should be 10 times higher than originally claimed.

If only 1% of pipelines are built in mountainous regions, the scaling factor should be 100.  This indicates the importance of scaling factors.

A similar problem exists with tanker shipping.   Most of the time, tankers travel in open oceans, where the risk of running aground is effectively zero.   In contrast, the proposed Northern Gateway tanker shipping route is narrow, twisty, and hundreds of kilometers long, from the port at Kitimat out to Hecate Straight and the open ocean.  The worldwide tanker data must be scaled to account for this discrepancy.

In their report, DNV states that, on average, tankers spend 10% of their time in regions where they can run aground (QRA, p5-51).  As justification for this 10% number, they cite a report that they wrote in 2004 for a similar project on the St. Lawrence River.  This 10% number is then used to adjust the worldwide incident data by a factor of 10, as discussed in the example above.

No real justification for the choice of a critical risk parameter

Many intervenors in the Joint Review Panel, including CPE, questioned Northern Gateway and DNV about this reference.  Ecojustice, a large Environmental Advocacy Group, requested that Northern Gateway provide a copy of the report, in English, but Northern Gateway said none existed.

View PDF Rabaska 2004

Hoping to find a scientific justification for the choice of 10%, CPE found the French publication and translated the relevant sections (p73) into English.  As it turns out, the referenced document does not provide any analysis or justification for this number, but simply assumes, based on estimations, that it is correct.  This is a gross error in a report that claims to take a scientific approach to a problem.

Typically, in a scientific analysis, all figures would be presented with some kind of justification for their validity.  This is an important part of the scientific process as it allows others to repeat the analysis or experiment and verify (or disprove) the original findings.  When a figure is presented with no justification, we are forced to question its validity as we are left with nothing to rely on except the credibility of the authors.  When they are in the employ of the proponent there is a clear conflict of interest, another issue that must be explicitly declared by authors wishing to publish in credible scientific publications.

The scaling factors were chosen in a closed process and no records were kept

Once a per-nautical mile incident frequency number is determined, it has to be scaled to account for “local factors” (QRA, p5-51).  In order to determine these scaling factors, DNV conducted a hazard identification (HAZID) process (QRA, p 4-40), which consisted of a closed one-day meeting to gather the opinions of 7 ship pilots and mariners who “had experience piloting and conning vessels to and from terminals in Kitimat and working on marine projects along the BC coast.”

The participants were asked to identify hazards that could cause a tanker incident, and were then asked (QRA, p4-42) to assess the frequency and the consequence of each hazard.  These numbers were then used to produce overall risk scaling factors for each segment of the proposed routes (QRA, p5-52).

View PDF Quantitative Risk Analysis

No documentation of the process

The scaling of worldwide incident frequency data to the BC coast is critically important to the overall calculated risk of a tanker spill.  As such, we would expect that this part of the risk analysis would be thoroughly justified and described in detail.  However, after completing the HAZID process, DNV convened a second group of Norwegian experts, in Norway, on May 19th, 2009, and somehow translated the subjective findings of the HAZID process into scaling factors (p5-52). They provided no records or methodology for how the scaling factors were produced from the original HAZID numbers (March 19, 2013 JRP hearings, line 31766).

View PDF March 19, 2013 JRP hearings

This process cannot be considered scientific, empirical, quantitative, or objective as it relies on the subjective assessments of interested parties (coast pilots, who will be employed by the tankers) and the undocumented manipulation of data, without any described methodology, by a paid consultant of the proponent.

As such, we are forced to heavily discount the reliability of this study as a subjective exercise and we propose that it bears little utility in assessing the overall risk of a tanker spill.

The QRA does not account for increased traffic from liquefied natural gas development

01_LNG-Traffic-increases-risks-embed-01

One of the most striking shortfalls of the QRA is the lack of a thorough accounting for an increase in tanker traffic in the study area.

The QRA describes the traffic densities in the study area as relatively low compared to world averages (QRA, p5-61).  While this may have been the case when the QRA was written in 2010, there have been significant changes to the expected traffic density since then.  The government of British Columbia is a strong proponent of the natural gas industry and tracks LNG projects, proposed and approved, on its LNG in BC website (link below).   We have added up all of the proposed and approved projects for the Kitimat terminal based on this information.

BC Government LNG in BC website:  http://engage.gov.bc.ca/lnginbc/lng-projects/

Screenshot 2014-03-10 19.08.41

From these numbers it is possible to estimate the expected LNG tanker traffic to the port of Kitimat.  An average LNG tanker carries approximately 155,000 cubic meters of LNG and a cubic meter of LNG weighs, approximately, 450 kg.  An average LNG tanker therefore carries 155,000*450 = 69,750,000 kg or 69,750 metric tonnes of LNG.

The total expected LNG tanker traffic is therefore 38,100,000 / 69,750 = 546 tankers per year.   It is reasonable to assume that not all of these projects will be approved, so we assumed only 430 tankers per year, approximately 25% less than the calculated amount.

Adding this number to the expected 220 tankers per year carrying diluted bitumen for the Northern Gateway project produces total expected tanker traffic of 650 tankers per year.

Traffic density strongly impacts the risk of a collision

According to the QRA, the probability of a collision increases with the square of the traffic density (QRA, p5-61).  If you double the traffic, the probability increases by a factor of 2×2, or 4. If you triple the traffic density, the probability of a collision increases by a factor of 3×3, or 9.

It is easy to see why an accurate understanding of the projected traffic density is important to the risk analysis: small increases in the projected traffic can have a dramatic impact on the risk of a collision.

During questioning in the JRP hearings, Enbridge’s experts (March 19, 2013: line 31994, 32034) pointed out that, according to the QRA, the risk of a collision was a small overall component of the risk and that, even if it were to increase, it doesn’t have a significant impact on the overall risk.

View PDF March 19, 2013 JRP hearings

We disagree with this statement.

Increasing traffic from 220 tankers per year to 650 is an increase of 2.955.  2.955×2.955 = 8.732.

The collision risk is therefore nearly 900% larger than what DNV originally calculated.

In order to figure out how the increased risk of a collision impacts the overall probability of a spill, we deconstructed and recalculated the risk numbers in the DNV report and found that the unmitigated overall probability of a spill greater than 5,000 cubic meters (a $1.2 billion spill) is actually 34% higher when updated traffic numbers are taken into account.

Effectiveness of mitigation is not proven

Central to Northern Gateway’s risk analysis is the idea that the use of tugs provides a significant reduction in risks.  The consultants who were paid by Northern Gateway to produce the risk analysis claim that the use of tugs can reduce the frequency of grounding by 80-90%.

Table 8-1 from QRA

Table 8-1 from QRA

QRA

Figure 8-1 from the QRA

The mitigating effect of tugs is critical to the overall risk of a spill, as shown in the graph of spill ‘return periods’ (QRA, Figure 8-1, page 8-125). Looking at figure 8-1 from the QRA, the use of tugs increases the return period of a spill by between 57% and 316%. However, under questioning during the Joint Review Panel process, DNV experts were unable to provide any documentation or justification for these numbers.

During the April 4, 2013 hearing in Prince Rupert, Cheryl Brown, from the group Douglas Channel Watch, questions Auden Brandsaeter, the lead analyst who produced the Quantitative Risk Analysis, about the justification for the 80-90% risk mitigation. Mr Brandsaeter states that these numbers were from a study conducted by another group, for another client (the Fawley study). When asked whether that information can be made available for public scrutiny, Mr Brandsaeter states that the report is confidential. From line 4203:

View April 4, 2013 JRP hearings PDF

Screenshot 2014-03-10 13.55.23

Importantly, the Fawley study was not based on actual, historical data.  It was simply an analysis of the problem, with calculations, estimations and more expert testimony.  In other words, there is no data to actually confirm that the analysis was accurate.  Furthermore, Steven Scalzo, from a tug company, when asked about whether they’ve kept a databank of actual incidents, says no, from line 4242:

Screenshot 2014-03-10 13.57.32

Screenshot 2014-03-10 13.57.45

DNV and Northern Gateway continue to point to the fact that there have not been any major incidents with the use of tugs.  This is a welcome fact (if it is true) but it misses the important point that, for rare events, this does not confirm that the analysis is correct.  Furthermore, there are no data to document whether the use of tugs is as effective in this particular situation, where the narrow channels may constrain the movement and operation of tethered tugs using tow lines.  The proponents have simply not provided any real-world, scientifically verifiable data to back up their claims.

The way in which the QRA presents its results is misleading

We disagree with the appropriateness of DNV expressing risks in terms of return periods. In fact, when all their mitigating factors are taken into account (use of tugs, scaling factors, etc.), their return periods for a spill greater than 5,000 m3 is 550 years. If the reduction factor for the use of tugs is not introduced, the return period for the same spill magnitude is shortened to 250 years. This may appear to be a long time, but what is not explained is that a return period is, by definition, an average time between spills.

In fact, given that just one spill of this magnitude would likely produce a catastrophic event, what is important is to assess the probability of at least one spill occurring within the 50 year operating life for the project. This probability can be mathematically calculated from the return period using Poisson’s Arrival Theorem. Thus, for a 550 year return period, the probability of at least one spill greater than 5,000 m3 during the 50 year life is almost 9%.

This is an optimistic number, given that it incorporates the estimated effect of the use of tugs. If the 250 year return period is used, the probability increases to about 22%. This more realistic way of looking at the DNV calculations clearly shows that the probabilities are clearly unacceptable, considering that most infrastructure projects are designed to much lower tolerable probabilities of failure. Of course, if there is a spill probability of 9%, one could argue that there is a 91% probability of no spill. Thus, the operation of the project might be lucky throughout its economic life. But depending on just luck should not be a scientific way of concluding that the project is safe.

Risk is not only determined by probabilities or return periods, but as a combination of probability and the magnitude of the consequences of a disaster.  Since the QRA does not provide an estimate of the consequences, concentrating only on the return periods of spills, we conclude that , in the final analysis, risk has not been assessed in the QRA.

Overall, a deeply flawed and unscientific analysis

03_Understanding risks-embed-01

Throughout our analysis of  the DNV QRA, we continually came across the same problem: a systemic lack of credible, scientifically verifiable evidence or methodology to back up the claims made by DNV and Northern Gateway.  If the QRA were a scientific document, it would contain the necessary reference information required for independent researchers to verify all of the claims, assumptions, and methods employed by the authors.  Instead, the trail turns cold after one or two levels of investigation.

For example, when trying to validate the claim that tankers spend 10% of their time in confined areas where grounding may occur, we looked up the referenced document (Rabaska 2004), translated the text from the original French, and found that DNV had in fact not performed a rigorous analysis to arrive at the 10% number.    To reference this assumption in a second publication is equivalent to backing up assumptions with assumptions.

To address some of these concerns, we attempted to  engage in a substantive debate with Northern Gateway, as an intervenor in the Joint Review Panel process, to investigate some of these issues and seek further evidence and justification for their claims.  Our analysis was dismissed in an off-hand manner, again with no reference to scientifically verifiable statements.  In other words, our criticisms were ignored.

When taken as a whole, the sum of unjustified  claims, inconsistencies, and inaccuracies leave us with a deep sense of concern for the validity and utility of this study.    We are forced to conclude that the QRA bears no meaningful results and that the risks of a significant tanker spill from the Northern Gateway project remain unknown.

What does the Joint Review Panel have to say?

View PDF JRP Report Vol. 1

View JRP Report Vol. 2

In its final report approving the Northern Gateway project, the Joint Review Panel states many times that they were required to weigh the costs and benefits of the project in a science-based manner.  They were “required to determine the sufficiency of the application … and conduct a technical analysis of the project based on all of the evidence.” (vol 1, p8)  They state that their role was to “conduct an independent, science-based, open, and respectful hearing process.”   (vol  1, p8) “Science and law provided the framework for [their] hearing process…” and they “considered the evidence in a careful and precautionary manner…” (vol 1, p11)

We have thoroughly reviewed the sections of the JRP’s final report and the 209 conditions that pertain to the tanker shipping component of the project and we are disappointed to find that none of our concerns have been addressed in a meaningful way, either by Northern Gateway, the Panel’s findings, or its conditions.

Unclear how the Joint Review Panel made its decision

The JRP dismisses our concerns, and the concerns of many other intervenors, with an array of confusing and obscure statements about their methodology and their understanding of how risk assessments are employed:

“The Panel’s view on the likelihood of a large marine shipping spill is not based on a specific number that attempts to provide an absolute indicator of the probability of a spill event. The Panel is of the view that it would not be appropriate to do so. The Panel is of the understanding that marine shipping risk assessments, such as Northern Gateway’s quantitative risk analysis and the federal government’s ongoing Canada-wide risk assessment for oil spills from ships, are intended to provide an indication of spill return periods or probabilities based on potential hazards, and to inform mitigation to address such hazards. These risk assessments are often conducted in the context of existing marine shipping.” (vol 2, p147)

“The Panel finds that Northern Gateway’s approach to assessing risk is appropriate and that it was properly applied. The Panel sees little practical value in an alternative approach that would attempt to derive a hypothetical risk number as a fundamental decision point that indicates whether the project should proceed.” (vol 2, p148)

We are unsure what to make of statements like these, in light of the Panel acknowledging at the beginning of the report that they were “required to conduct a technical analysis of the project based on all of the evidence.” (vol1, p8)  In order to conclude that the benefits of the project outweigh the risks and costs, we presume the Panel utilized some measure of the risks.

“The Panel finds that a large spill, due to a malfunction or accident, from the pipeline facilities, terminal, or tankers, is not likely.” (vol 2, p168)

For the marine component, it appears from their report that the Panel accepts the risk estimates that were put forward by Northern Gateway via the Quantitative Risk Analysis performed by DNV:

“The Panel does not accept parties’ arguments that the quantitative risk analysis was improperly prepared or subject to substantial flaws.” (vol 2, p147)

And:

“The Panel finds that Northern Gateway’s approach to assessing risk is appropriate and that it was properly applied. The Panel sees little practical value in an alternative approach that would attempt to derive a hypothetical risk number as a fundamental decision point that indicates whether the project should proceed.” (vol 2, p148)

Our reading of the above statements leads us to conclude that, even though the JRP is not basing its decision on a single risk number, they accept the risk estimations put forward by Northern Gateway as being acceptable.

Their dismissal of our concerns, along with those of many other intervenors, is an indication to us, in the absence of any meaningful reply, that they do not understand our arguments, that they have already made up their minds, or that they, in fact, are using a separate process for evaluating the risks of the project – one that is not scientific or based on a technical analysis of the evidence.