# Expectation and Prediction In this document, we explain the three main types of predictions in our system, how our scoring system works based on them, and how we combine them. ## Prediction versus Expectation In our web application, a prediction (also referred to as a forecast or expectation) can reflect either the output of a statistical model or a user's own subjective belief about a future value. While these sources differ—models rely on data and algorithms, whereas users draw on intuition or domain knowledge—the way we represent their expectations is unified. Whether the forecast comes from a model or a person, it can be expressed as a single value (point prediction), a full probability distribution (to reflect uncertainty), or a simpler directional statement (e.g., the probability of an upward or downward movement). This consistent format allows our system to treat both types of inputs equally. At Arenil.com, we aim to provide flexible tools that allow users to easily express their subjective expectations. Consistent with the ideas discussed above, our goal is to ensure that users can communicate their views in intuitive and natural ways, while still grounding those expressions in rigorous statistical interpretation. For clarity and simplicity, we use the terms prediction, forecast, and expectation interchangeably throughout this document to refer to any expression about a future outcome. ## Types of Predictions There are three main types of predictions in our system: - **Point Prediction:** A single numerical value that represents the forecasted outcome. - **Direction Prediction:** Specifies the probability that the outcome will move in a certain direction—either upward or downward. - **Distribution Prediction:** Describes the forecast as a full probability distribution, capturing both the central tendency (such as the median or mean) and the uncertainty (variability) around the outcome. In addition to the primary prediction types, we also derive related information automatically: - **Direction from Point Prediction:** We infer the direction by comparing the predicted point to the most recent known value. - If the predicted point is greater than the last known value, we assign a probability of 1 to an upward movement. - If it is lower, we assign a probability of 1 to a downward movement. - If it is equal, we assign full probability to no change. - **Point from Distribution Prediction:** We use the **median** of the distribution as the representative point prediction. The median provides a robust central estimate that is less sensitive to outliers. - **Direction from Distribution Prediction:** Once the median is computed, we infer the direction using the same rule as for point predictions—by comparing the median to the last known value. Each prediction is converted into a standardized string format, depending on its type: - **Point Predictions:** Simply a numerical value. For example, `125.45`. - **Direction Predictions:** Represented as a pair of comma-separated probabilities. For example, `0.65,0.35` (Meaning: 65% chance of an upward movement and 35% chance of a downward movement.) - **Distribution Predictions:** Specified by the distribution name followed by the parameter list within parentheses. For example `normal(125.45,10.5)` or `empirical(120,125,130)`. Our application supports the following types of distributions for representing uncertainty: - **Normal Distribution:** The normal distribution is characterized by mean and standard deviation. This is written as `normal(mean, standardDeviation)` - **Empirical Distribution:** An empirical distribution is a more flexible way of expressing ideas about the future. This empirical distribution is defined by a strictly increasing array of quantiles. The distribution uses these quantiles to establish a uniform probability spacing between a lower bound (defaulted to 0.001) and an upper bound (defaulted to 0.999), effectively determining the probability interval for each successive quantile. It is represented as `empirical(q1,q2,q3,...)`. ## How Scores Are Calculated After the actual outcome is observed, we calculate several scores to assess prediction performance. ### 1. Direction Prediction Scoring The [Brier Score](https://en.wikipedia.org/wiki/Brier_score) is calcuated using the following formula:

\[ \text{Brier Score} = \frac{1}{3} \left[ \left( p_{\text{up}} - o_{\text{up}} \right)^2 + \left( p_{\text{down}} - o_{\text{down}} \right)^2 + \left( p_{\text{constant}} - o_{\text{constant}} \right)^2 \right] \]

where \\(p\_{\text{up}}\\), \\(p\_{\text{down}}\\), and \\(p\_{\text{constant}}\\) are the forecast probabilities and \\(o\_{\text{up}}\\), \\(o\_{\text{down}}\\), and \\(o\_{\text{constant}}\\) are the observed outcomes (1 if the event occurred, otherwise 0). A lower Brier score indicates a better, more calibrated forecast. ### 2. Point Prediction Scoring *Absolute Error* measures the difference between the predicted point and the actual value:

\[\text{Absolute Error} = |\text{Actual Value} - \text{Predicted Point}|\]

This metric is straightforward and useful when comparing performance within the same context, especially when the target variable (i.e., the variable being predicted) has a consistent scale. However, a major drawback of absolute error is that it depends on the *unit* of the data. An error of 10 might be significant in one application but negligible in another. *Absolute Percentage Error* represents the absolute error in percentage terms:

\[ \begin{aligned} \text{MAPE} = \frac{|\text{Actual Value} - \text{Predicted Point}|}{|\text{Actual Value}|}& \times 100\%,\\& \text{Actual Value} \ne 0 \end{aligned} \]

This metric helps in understanding the *relative* size of the forecast error. It allows for comparison across different datasets or scales, regardless of units. However, it can be unreliable when the actual value is very small or near zero, as it may result in extremely large or misleading percentage errors. ### 3. Distribution Prediction Scoring *Continuous Ranked Probability Score* (CRPS) evaluates the accuracy of the entire predictive distribution. A lower CRPS indicates a forecast that more accurately reflects the observed data. The CRPS for a predictive CDF \\(F\\) evaluated at an observed value \\(x\\) is given in the following:

\[ \text{CRPS}(F, x) = \int_{-\infty}^{\infty} \bigl(F(y) - \mathbf{1}_{\{y \ge x\}}\bigr)^{2} dy \]

where \\(\mathbf{1}\_{\{y\ge x\}}=1\\) for \\(y \ge x\\) and 0 otherwise. For a **normal distribution** with forecast mean \\(\mu\\), standard deviation \\(\sigma\\), and observed value \\(x\\), CRPS is calculated as:

\[ \text{CRPS}(x, \mu, \sigma) = \sigma \left[z \left(2\Phi(z)-1\right) + 2\phi(z) - \frac{1}{\sqrt{\pi}}\right] \]

where \\(z = \frac{x - \mu}{\sigma}\\), \\(\phi(z)\\) is the standard normal probability density function, and \\(\Phi(z)\\) is the standard normal cumulative distribution function. In our implementation, when \\(\sigma = 0\\), the CRPS calculation defaults to the absolute error, i.e., \\(|x - \mu|\\). For an **empirical distribution** with increasing quantiles \\(q\_1, q\_2, \ldots, q\_m\\), we approximate the CRPS integral using a Riemann sum:

\[ \text{CRPS} \approx \sum_{i=0}^{N} \bigl(F(y_i) - \mathbf{1}_{\{y_i \ge x\}}\bigr)^2 \, \Delta y, \quad y_i = \ell + i\,\Delta y, \quad \Delta y = \frac{u - \ell}{N} \]

where \\(\ell = q\_1 - h\\), \\(u = q\_m + h\\), and \\(h = (q\_m - q\_1)/10\\). We divide the interval \\([\ell, u]\\) into \\(N\\) equal subintervals of width \\(\Delta y\\), and on the \\(i\\)th subinterval \\([y\_i, y\_{i+1}]\\), we use the left endpoint \(y\_i\\) as a representative point. In most cases, the result is generally not sensitive to the exact choice of \\(\ell\\) and \\(u\\). Beyond a few ranges below the minimum or above the maximum quantile, the integrand \\((F(y) - \mathbf{1}\_{\{y \ge x\}})^2\\) becomes negligibly small. ## Combining Predictions Our system aggregates multiple forecasts into a consensus-based prediction. The combination method depends on the type of prediction: - **Combining Direction Predictions:** The probabilities of upward and downward movement are averaged across all forecasts to produce a new, combined direction prediction. - **Combining Point Predictions:** The individual point predictions are pooled and used to construct a distributional forecast. - **Combining Distribution Predictions:** We aggregate the individual empirical distributions into a single consensus distribution by taking a weighted linear combination of their cumulative‐distribution functions.