Back to top

Uncertainty

A model can never perfectly represent a natural system. Due to paucity of data, errors in system definitions and data used as well as limited system understanding, uncertainty us inherent in all MFAs. Data for material flow analysis originate from different sources and vary in terms of availability and quality, particularly if material stocks and flows of large-scale systems, such as regions or whole economies, are investigated. Ignoring uncertainty in MFA has raised doubts about the reliability of MFA results. Approaches to uncertainty analysis aim at making uncertainties transparent and reducing them. They enable the users to make more robust assumptions and to become aware of the model’s strengths and limitations. A good uncertainty analysis addresses both systematic errors (e.g., system definition) as well as random errors (e.g. data). There are different ways to deal with uncertainty.

Please note: The following text is taken from the existing MinFuture reports "A systems approach for the monitoring of the physical economy" and "Compilation of uncertainty approaches and recommendations for reporting data uncertainty". Please click on the afore-mentioned links to access and read the full publications.

 

Material flow analysis (MFA), is a term used by a wide range of approaches to describe material and energy stocks and flows in systems defined in space and time. A model can never perfectly represent a real system. Because of that, model predictions are always uncertain. Besides MFA concerns gathering, harmonizing, and analysing data about physical flows and stocks from different sources with varying qualities, therefore data limitations are unavoidable. Although studies of material flow systems can provide useful information, they also depend on data and information and their absence can be a limiting factor. In addition, results of limited or unknown accuracy may have negative impacts on subsequent decisionmaking processes. Clearly, if MFA is seen as a way of compiling data to create information about material stocks and flows and to aggregate this information to create knowledge about material flow systems, the quality of its fundamental components, data, is crucial.

Read More

Methods to deal with uncertainty in MFA range from qualitative discussions to sophisticated statistical approaches. Different methods will be evaluated. The majority of methods can be classified into four groups:

  • Data classification methods
  • Uncertainty analysis approaches
  • Sensitivity analysis approaches
  • Comparisons of model structures

 

The nature of uncertainty in MFA
In MFA, uncertainty analysis should consider all available information about the system and data and reflect its purpose and data quality. The uncertainty of the data and the accuracy of the results are fundamental derivatives of the evaluation process. As MFA concerns gathering, harmonizing and analysing data about physical stocks and flows from various different sources with varying quality, limitations of data are unavoidable. The majority of data used in MFA are empirical quantities with uncertainty arising from different sources. The causes, sources and types of uncertainty are summarised in Table 1, Table 2 and Table 3 below.

Table 1: Causes of Uncertainty

 

Table 2: Sources of uncertainty

 

Table 3: Types of uncertainty

Ways to deal with uncertainty
Statistical analysis uses inaccurate data samples to get as decent as possible knowledge of one entity. If there is enough data available, it is possible to use statistics (median, standard deviation). In some situations, however, problems may occur with statistical methods, therefore, different approaches to treat uncertain data have been developed. Methods to deal with uncertainty in MFA range from qualitative discussions to sophisticated statistical approaches, see Table 4 below.

Uncertainty is often characterized without the use of formal procedures, which impairs statements about the reliability of the MFA results based on uncertainty analysis. Therefore, consistent and transparent procedures are imperative for uncertainty analysis in MFA. In addition, different uncertainty types need to be addressed by different concepts used to express and propagate uncertainty.

A systematic iterative procedure for handling uncertainty in MFA is presented in Figure 14. Depending on the specific application, only parts of the scheme may be completed. Some studies may not warrant full uncertainty analysis because of different emphases of the MFAs. If the MFA is mainly used to quantify material balances to improve the database, steps 1 to 3 are of major importance and step 4 resembles the uncertainty result. If the goal of the MFA does not require describing the inherent uncertainty of model outputs resulting from the data used, then the characterization of data uncertainty (step 2) is less important, but sensitivity analysis and/or scenario modelling (step 5) are important elements to be taken into consideration.

 

Table 4: Approaches to deal with Uncertainty

 

Figure 8: Schematic illustration of a systematic procedure for uncertainty analysis in MFA

The schematic framework in Figure 8 under Approaches above, should be considered when incorporating uncertainty analysis in MFA, but could also be used to justify the approach followed for quantifying uncertainty in a study. In the pyramid, uncertainty is associated with all of its components as illustrated below.

  • System: A model is a simplified version of the real system, which is difficult to replicate perfectly and therefore uncertainty is inherently present. The first step for handling uncertainty in MFA is to define the elements of the system and the mathematical relationships between them in consideration of the mass balance principle.
  • Data: Because data often originates from different sources, it is unavoidably of varying quality. To deal with uncertain data appropriately, functions need to be characterised. If sufficient empirical evidence is available, statistical parameter estimation techniques or goodness-of-fit tests can be applied.
  • Model: Assumptions and simplifications are made that lead to uncertainty regarding the validity of the model predictions to reflect real world situations. Usually, model equations are solved numerically and therefore associated uncertainty is low. For exploratory MFA, sensitivity analysis is used to evaluate the effects of parameter variation on the model outputs. It forms the basis to identify critical model parameters, which can influence the material flows in the system and therefore the final outcomes.
  • Indicators: The calibrated model, where satisfying agreement between data and model results has been achieved, is used to calculate the model outputs (material flows and associated uncertainty). Indicators and their associated uncertainty can therefore be calculated. It is also possible to interpret uncertainty estimates for the resulting flows.
  • Visualisation: It is possible to visualise uncertainty in many different ways, depending on the kind of visualisation of the MFA itself (e.g. resolution), the related results (e.g. Sankey, Pie, paired bar, maps, stacked column) and the available data.
  • Decision-making: Although studies of material flow systems can provide information and convey knowledge, they also depend on information and knowledge in their production process. A lack of useful information can be a limiting factor about the level of detail provided in an analysis with subsequent influence on decisions to be taken when considering such result. New challenges emerge when the qualitative discussions of political science meet the quantitative approach of physical science. There seem today to be little support for how few, subjectively estimated data with large uncertainties should be taken into account. Still, there is a need to consider and calculate results from uncertain data. It is important therefore to convince data providers to include data uncertainty in their publications. In addition, often decision-making, whether this is policy or strategy development have a broader scope and focus and may account for multiple parameters that have not been taken into consideration in the model development (e.g. social impacts, job opportunities etc). In an ideal scenario, the scope of a study should be informed about parameters of influence that are beyond the scope of MFA to account for uncertainty accordingly.

In conclusion, there are a handful of applicable approaches to consider data uncertainty in MFA. The employment of MFA software would facilitate the implementation of these approaches and reduce the additional workload. However, such software support is not yet
on the market (with the exemption of STAN3, which is limited to normally distributed values) and there is a strong necessity to fund such development. Only then, MFA could enter into an era where reporting uncertainty ranges of stocks and flows is integrated during
the study development stage. This would help to judge or gauge the reliability of MFA studies and allow comparative studies to take place.

  • The systematic evaluation of uncertainty in MFA is important to understand the robustness of material flow estimates, independent of the approach used for expressing uncertainty.
  • Consistent and transparent procedures for uncertainty characterization are imperative for uncertainty analysis in MFA.
  • Uncertainty should be seen as a red thread throughout the modelling study starting from the very beginning.
Impression from the third MinFuture workshop in Brussels, showing the participants listening to an input presentation
Event

The purpose of this workshop was to further develop the roadmap for monitoring the physical economy. During the spring of 2018, the MinFuture project has held several commodity specific workshops to test the developed framework and to identify commodity specific trends, opportunities and challenges that can inform the MinFuture roadmap.  Workshops has been held on aluminium, cobalt, neodymium, platinum, phosphorus and construction aggregates, and stakeholders from different parts of the supply chain has contributed to it.

Event

This workshop is part of an EU Horizon 2020 project MinFuture (Global material flows and demand-supply forecasting for mineral strategies...

Event

This workshop is part of an EU Horizon 2020 project MinFuture (Global material flows and demand-supply forecasting for mineral strategies; see details in the flyer attached or the project website: http://minfuture.eu/).

Result

The MinFuture workshop synthesis brief describes the main insights from discussions on:

  • How can we add more relevance and credibility to data published on raw materials? What context is missing that might enhance their status? How could we present data using a systemic MFA? perspective
  • How do raw material data reporting schemes (information flows) currently operate at national, regional and global level?
  • What raw material indicators are often used to identify issues with raw material supply/ demand? What are their strengths and weaknesses and how do they relate to material flow analysis?
Event

Aimed at developing a common framework to analyse global mineral raw material flows, which can be agreed and used at international level, the MinFuture project intends to support data collectors, providers and users. Improving knowledge and quality of data on material cycles was found to be essential, but is faced with challenges such as interrupted information flows or lacking data availability. The first MinFuture Workshop (‘Methodology workshop’) served to discuss how MinFuture could support key data providers and users.

Group photo of the participants of the 2nd MinFuture workshop
Event

The purpose of this workshop was to initiate a dialogue with key stakeholders that report raw materials data, use data to develop MFA models, or use MFA models to inform decision making. The knowledge and needs of data providers, users and decision actors are different, but in order for a ‘common approach’ to be developed their input is required.

Result
MinFuture Deliverable D3.1

The brief presents key discussion items and main findings from the June 2017 Workshop in Vienna. In order to tackle challenges such as insufficient information flows or lacking data availability, a (more) systemic understanding of global mineral raw material flows is needed. Mapping the system context and making data/information gaps explicit will help identifying possible improvements.

Pages

Subscribe to Uncertainty