marti.ai tech blog

Combining Professional Forecasters with Copulas (Part 3): A Walk-Forward Horserace

Sun, 12 Jul 2026 09:00:00 +0000

This is the third post in the series. Part 1 introduced the Survey of Professional Forecasters data. Part 2 developed two copula-based ways to combine point forecasts. This post asks how those ideas behave in a walk-forward empirical test.

The empirical results do not identify a single dominant copula method. In this quarterly macro setting, the strongest methods are mostly parsimonious. The nowcast is hard to beat. Error GLS and shrunk Error GLS perform well because they concentrate on the most informative horizon while correcting for redundant and correlated forecast errors. More flexible copula methods are informative diagnostics, but in this sample their additional flexibility does not translate into better out-of-sample forecasts.

From the first two posts to this horserace

Part 2 separated two probability questions.

The outcome-forecast approach models the joint distribution of the realised outcome and the forecasts:

\[(Y, X_1, \ldots, X_D).\]

Its question is:

\[\mathbb{E}[\text{outcome rank} \mid \text{forecast ranks}].\]

The forecast-error approach models the dependence structure of the forecast errors:

\[E_k = X_k - Y.\]

Its question is:

\[\text{which candidate } y \text{ makes } (X_1-y,\ldots,X_D-y) \text{ plausible?}\]

These two questions can lead to similar linear formulas under restrictive Gaussian assumptions, but they are not the same statistical object. The distinction matters once we evaluate rank skill, level accuracy, nonlinear dependence, and different point summaries of a likelihood.

The target is now an official realised value

Parts 1 and 2 used an SPF-aligned proxy target constructed from the available survey panel. For the final horserace I rebuild the panel from the Philadelphia Fed’s official SPF evaluation files. This gives a more precise target definition.

For each variable, the panel contains three realised values:

Target	Meaning
`target_initial`	first official realised value used for real-time evaluation
`target_latest`	latest revised official value
`target_spf_proxy`	earlier proxy target used in Parts 1 and 2

The main experiment uses target_initial. This is closest to the real-time question: what would the forecast have predicted before later data revisions?

I keep target_latest for a revision-sensitivity check. This matters because macro realised values are revised. A forecasting method can look different depending on whether we score it against first releases or later vintages.

The official-target panel covers RGDP, NGDP, PGDP, UNEMP, CPI, INDPROD, and HOUSING. I exclude CPROF from the main horserace because the same aligned official evaluation structure is not available, and the CPROF definition changes over time.

The figure shows why this matters. For unemployment, the different target definitions are almost identical in rank. For growth variables, especially RGDP, the proxy, initial release, and latest revised value can differ materially.

Evaluation design

The main experiment is a walk-forward expanding-window evaluation.

For each target date, a method is trained only on information from earlier target dates. The training window then expands by one date and the procedure repeats. This is closer to the way a forecast-combination method would be used in practice than a single fixed train/test split.

The cross-horizon experiment combines SPF consensus forecasts made at leads 0 through 4 for the same target. Lead 0 is the nowcast. Longer leads are progressively earlier forecasts for the same realised quarter.

The methods are:

Family	Methods
Simple benchmarks	persistence, Lead 0, equal weight, trimmed mean, inverse MSE
Outcome-forecast dependence	OLS on raw ranks, Gaussian outcome copula, shrunk Gaussian outcome copula, truncated outcome vine
Forecast-error dependence	Error GLS, shrunk Error GLS, Gaussian error copula, vine error copula

The rank metric is computed as follows. Actuals and predictions are ranked separately within each macro variable, then those within-variable ranks are stacked and one Spearman correlation is computed. This avoids comparing the level of unemployment directly with the level of GDP growth, while still giving one summary number for the full panel.

The level metric is scaled RMSE. Each error is divided by the standard deviation of the realised target in the training sample for that variable. This makes the level errors roughly comparable across variables with different units. Scaled bias is the average scaled error, so positive values mean the method tends to overpredict.

The main result

The main table is below. Higher rank correlation is better. Lower scaled RMSE is better.

Method	Stacked rank rho	Scaled RMSE	Scaled bias
Lead 0	0.788	0.461	-0.005
Shrunk Error GLS	0.786	0.459	-0.007
Error GLS	0.778	0.461	-0.004
Gaussian outcome copula	0.766	0.647	0.014
OLS raw ranks	0.766	0.788	0.054
Gaussian error copula, flat mean	0.757	0.612	0.033
Vine error copula, flat mean	0.755	0.623	-0.003
Inverse MSE	0.716	0.689	0.043
Truncated outcome vine	0.702	0.746	-0.087
Equal weight	0.634	0.822	0.061
Persistence	0.598	1.179	0.017
Trimmed mean	0.573	0.919	0.067

The persistence benchmark is the last observed realised value available in the expanding training sample. The strongest benchmark is not a sophisticated model. It is Lead 0, the nowcast. In this cross-horizon setting there is a structural skill hierarchy: the forecast made closest to the target quarter contains much more information than the forecasts made several quarters earlier.

The first three rows are very close. Lead 0 has the highest rank correlation, while shrunk Error GLS has the lowest scaled RMSE by a small margin. The nowcast and the two Error GLS variants are therefore broadly comparable at the top of the table. A paired quarter-block bootstrap gives the same interpretation: the gap between Lead 0 and shrunk Error GLS is small relative to sampling variation. Each row below is computed as the first method minus the second method. For RMSE, negative values favour the first method.

Comparison	Delta rank rho	95% bootstrap interval	Delta scaled RMSE	95% bootstrap interval
Lead 0 minus shrunk Error GLS	0.003	[-0.004, 0.009]	0.003	[-0.023, 0.029]
Lead 0 minus Error GLS	0.009	[-0.005, 0.027]	0.001	[-0.068, 0.058]
Lead 0 minus Gaussian outcome copula	0.021	[0.007, 0.035]	-0.172	[-0.367, -0.036]

The ranking against equal-weight is less ambiguous. Relative to equal-weight, Lead 0, shrunk Error GLS, Error GLS, and the Gaussian outcome copula all have positive bootstrap intervals for rank correlation and negative intervals for scaled RMSE.

The Gaussian outcome copula remains a strong rank predictor, but its level RMSE is worse. This level score combines two separate steps: estimating the conditional rank, then mapping that rank back to a level using the empirical training distribution of realised outcomes. The weaker RMSE can therefore come from the marginal calibration and inverse-rank mapping, even when the dependence model orders outcomes well.

Why Error GLS holds up

Error GLS estimates the covariance matrix of forecast errors,

\[\Sigma_E = \operatorname{Cov}(E_1,\ldots,E_D),\]

and uses its inverse, the precision matrix, to form

\[\hat y = \frac{\mathbf{1}^{\top}\Sigma_E^{-1}x} {\mathbf{1}^{\top}\Sigma_E^{-1}\mathbf{1}}.\]

This formula has a simple interpretation. Forecast directions with high error variance count less. Forecast directions that duplicate the same error information also count less. When several horizons are highly correlated, Error GLS avoids treating them as independent pieces of evidence.

The estimated weights make this visible. Averaged across the walk-forward evaluation, both Error GLS variants put most of the weight on the nowcast. The weights sum to one, but they are not constrained to be positive: Error GLS is correcting correlated forecast errors, not imposing a convex average. The unshrunk estimator uses larger offsetting weights. The shrunk version keeps the same broad pattern, but pulls the secondary weights closer to zero; some average weights can still be negative.

Method	Lead 0	Lead 1	Lead 2	Lead 3	Lead 4
Error GLS, average weight	1.370	-0.352	0.005	-0.136	0.112
Shrunk Error GLS, average weight	1.073	0.007	-0.062	-0.056	0.039

This is also why Error GLS ends up close to Lead 0 in the leaderboard: it mostly learns to trust the nowcast, while using the other horizons as smaller error-correction terms.

This is a relatively low-dimensional object to estimate. A full KDE plus error-copula method is more demanding: it estimates each error marginal distribution, fits a multivariate dependence model, and then solves a one-dimensional prediction problem for each new forecast vector. In this sample, the extra flexibility comes with enough estimation variance to offset much of its potential benefit.

Error-copula point summaries

Part 2 introduced the forecast-error copula through the likelihood of the implied error vector:

\[f_E(X_1-y,\ldots,X_D-y).\]

The most direct point forecast is the likelihood mode:

\[\hat y_{\mathrm{mode}} = \arg\max_y f_E(X_1-y,\ldots,X_D-y).\]

That answers the “most plausible error vector” question. But it is not the only possible point summary. If the evaluation criterion is squared error, a mean of the implied likelihood can be more natural than its mode.

I therefore compare three summaries:

Summary	Definition
Mode	choose the candidate $y$ with maximum error likelihood
Flat-prior mean	average candidate $y$ values using weights proportional to $f_E(X-y\mathbf{1})$
Outcome-prior mean	average candidate $y$ values using weights proportional to $f_E(X-y\mathbf{1}) f_Y(y)$

In this experiment, the flat-prior mean is the best of the flexible error-copula summaries. The outcome-prior mean does not clearly help, probably because it adds another estimated object, the marginal density of $Y$. The mode is conceptually clean, but it is not automatically the best summary for RMSE.

The comparison is similar for the Gaussian error copula and the vine error copula:

Error-copula method	Summary	Stacked rank rho	Scaled RMSE	Scaled bias
Gaussian	Mode	0.709	0.694	0.001
Gaussian	Flat-prior mean	0.757	0.612	0.033
Gaussian	Outcome-prior mean	0.745	0.643	0.065
Vine	Mode	0.714	0.703	-0.042
Vine	Flat-prior mean	0.755	0.623	-0.003
Vine	Outcome-prior mean	0.740	0.644	0.040

The error-copula likelihood ranks candidate outcomes by the plausibility of their implied error vectors. Turning that likelihood into a point forecast requires an additional choice: mode, mean, or another summary matched to the evaluation loss.

By-variable rank correlations

The table below reports Spearman rank correlations computed separately for each macro variable.

Variable	Lead 0	Shrunk Error GLS	Error GLS	Gaussian outcome copula	Gaussian error copula, flat mean	Equal weight
CPI	0.809	0.803	0.793	0.778	0.794	0.642
HOUSING	0.967	0.969	0.973	0.963	0.964	0.894
INDPROD	0.777	0.782	0.774	0.762	0.691	0.485
NGDP	0.693	0.682	0.650	0.616	0.674	0.493
PGDP	0.709	0.710	0.711	0.702	0.715	0.619
RGDP	0.571	0.561	0.554	0.550	0.511	0.336
UNEMP	0.997	0.995	0.994	0.992	0.961	0.964

The result is not driven by a single variable. The near-horizon forecast is strong almost everywhere. The two Error GLS variants are especially competitive for HOUSING, INDPROD, PGDP, and UNEMP, with shrunk Error GLS often close to the best method. The Gaussian outcome copula remains competitive as a rank predictor, but it is generally a little weaker than the nowcast/Error GLS group here, and its rank-to-level conversion performs less well for level accuracy.

This table is mainly a diagnostic for whether the pooled result is driven by one macro series. It is not meant to rank methods precisely within each variable, where several top correlations are very close.

RGDP remains difficult. Equal weighting performs poorly because it averages a relatively informative nowcast with much weaker longer-horizon forecasts. This is exactly the setting where dependence correction helps: the combination method should not treat all horizons as equally informative.

Initial releases versus latest revisions

The same predictions can be scored against initial releases or latest revised values. I use the initial release as the main target because it is the first official realised value available after the target period.

I also report the same scores against the latest revised value. This is a useful check because later vintages can reflect additional source data, benchmark revisions, or statistical reprocessing after the initial release.

The broad ranking is similar across the two targets: Lead 0 and Error GLS remain very strong. The level metrics move more than the ranks, especially for the methods closest to the nowcast. This is why I keep the initial-release and latest-revision results side by side.

Method	Rank rho, initial release	Rank rho, latest revised	Scaled RMSE, initial release	Scaled RMSE, latest revised
Lead 0	0.788	0.747	0.461	0.512
Shrunk Error GLS	0.786	0.741	0.459	0.520
Error GLS	0.778	0.737	0.461	0.531
Gaussian outcome copula	0.766	0.732	0.647	0.653
Equal weight	0.634	0.603	0.822	0.821

The deterioration is larger for the top nowcast and Error GLS methods than for equal-weight, because these methods are more tightly tied to the near-term official release. The ordering remains similar, but the target vintage changes the measured size of the advantage.

Rolling performance through time

For each target date, the figure computes one within-variable Spearman correlation using the previous 32 quarters of out-of-sample predictions. It is not a pure sample-size experiment. Later fitted forecasts have more training data available, but they are also evaluated in later macroeconomic periods.

Equal weight is a fixed rule, so its line does not move because of re-estimation. It moves because the evaluation window changes. For fitted methods, the rolling lines combine two effects: the training sample expands, and the forecast environment changes.

The main pattern is consistent with the pooled table. Lead 0 and Error GLS remain difficult to separate for much of the sample, while equal weight is usually below them. The gap is not constant through time, which is another reason to treat the single pooled leaderboard as a summary rather than a universal ranking.

Named-forecaster case studies

The cross-horizon experiment has a built-in skill hierarchy: lead 0 is usually more informative than lead 4. The classical forecast-combination puzzle is different. It is often about combining forecasters who make predictions for the same target at the same horizon, with similar information sets and similar skill.

I therefore keep a separate named-forecaster experiment. For CPI, RGDP, and UNEMP, at horizons 1 and 2, I form small panels of individual forecasters and run the same expanding-window evaluation. These variables give three different forecasting problems with enough overlapping individual respondents: inflation, real activity growth, and unemployment. Horizons 1 and 2 keep the comparison at a common forecast horizon while avoiding both the nowcast-dominance issue and the thinner coverage of longer horizons.

The figure uses three-forecaster panels. For each variable-horizon pair, the panels are the three highest-coverage forecaster trios selected before looking at test performance.

This named-forecaster check is deliberately smaller than the cross-horizon experiment. To get a usable history, I have to reuse many of the same high-coverage forecasters and target dates across panels. The table therefore gives a useful check on how the combination methods behave when the signals are individual forecasters rather than different lead times, but not many independent repetitions of the same experiment. The point is to contrast two empirical settings:

Setting	Main feature
Cross-horizon consensus forecasts	strong structural skill hierarchy across lead times
Named forecasters at the same horizon	closer substitutes with similar information sets

Across these three-forecaster panels, the ranking is much less decisive than in the cross-horizon experiment. The persistence row is the last observed realised value, not a combination of forecasters. The other rows combine the selected forecasters. I round these summaries to two decimals because the panels overlap and the sample is smaller.

Method or benchmark	Named-forecaster rank rho	Scaled RMSE
Persistence	0.51	2.20
Shrunk Error GLS	0.48	1.94
Error GLS	0.47	1.95
Inverse MSE	0.46	1.93
Trimmed mean	0.46	1.93
Equal weight	0.46	1.93
Gaussian error copula, flat mean	0.45	2.03
Gaussian outcome copula	0.39	1.97

I also reran the same construction with four and five forecasters per panel. Larger panels average more individual forecasts, but they leave fewer usable target quarters: every selected forecaster must have reported a forecast for the same variable, horizon, and target quarter. In this SPF sample, that common-overlap constraint makes larger named-forecaster panels quickly lose observations.

Forecasters per panel	Median OOS quarters per panel	Persistence	Inverse MSE	Equal weight	Shrunk Error GLS	Error GLS	Gaussian error copula
3	62	0.51	0.46	0.46	0.48	0.47	0.45
4	46	0.48	0.54	0.54	0.54	0.49	0.47
5	31	0.49	0.49	0.48	0.48	0.42	0.39

The panel-size check does not change the qualitative conclusion. Equal weight, inverse MSE, and shrunk Error GLS remain close to one another, while unshrunk Error GLS and the flexible error-copula do not clearly dominate as the number of forecasters increases. This is closer to the classical forecast-combination puzzle: when forecasters have similar information sets, estimated weights may not improve much on simple regularised averages. Larger panels leave fewer common forecast dates, so I read the table mainly as a robustness check on the named-forecaster exercise.

What I take from the horserace

First, the nowcast is a serious benchmark. In a cross-horizon SPF combination problem, beating Lead 0 is difficult because the nowcast already reflects much of the relevant information.

Second, dependence modeling is still useful. Error GLS and shrunk Error GLS are competitive because they estimate a small, interpretable object: the covariance matrix of forecast errors. That is enough to correct for noisy and redundant horizons.

Third, rank prediction and level prediction are not the same objective. The Gaussian outcome copula is a good rank predictor, but mapping ranks back to levels introduces an additional calibration problem.

Fourth, flexible copulas need enough stable information. The vine and KDE-based error-copula methods can represent richer dependence, but richer models are not automatically better in SPF-sized macro samples.

The empirical exercise preserves the main distinction from Part 2. The results do not select one method that is best in every setting. The copula framework is useful here because it makes two different probability questions explicit:

\[\mathbb{E}[\text{outcome rank} \mid \text{forecast ranks}]\]

is not the same object as

\[\text{the outcome whose implied forecast-error vector is most plausible}.\]

Once those two objects are separated, the empirical results are easier to interpret. In this implementation, the outcome-side methods first estimate ranks and then require a separate rank-to-level calibration step. The error-copula methods search over candidate outcome levels; for each candidate $y$, they compute implied errors $x_k-y$. The copula coordinates are empirical ranks of those implied errors relative to the training errors, while the marginal density terms are estimated with KDE. Error GLS is the parsimonious version of the error-side idea: it uses only an error covariance matrix. In this SPF horserace, the strongest error-side results come from Error GLS and shrunk Error GLS rather than from the KDE plus error-copula methods.

Combining Professional Forecasters with Copulas (Part 2): Two Approaches to Dependence

Wed, 08 Jul 2026 00:00:00 +0000

This is the second post in a three-part series. Part 1 introduced the Survey of Professional Forecasters data. This post builds the modelling framework: two ways to use dependence when combining point forecasts. Part 3 runs the walk-forward empirical horserace on official SPF evaluation targets.

Where this post fits

Part 1 described the raw material: professional forecasters report point forecasts for macro variables such as GDP growth, inflation, unemployment, housing starts, and industrial production at several horizons. For a fixed variable and target quarter, write

\[y_t = \text{realised outcome}, \qquad x_{0,t}, x_{1,t}, \ldots, x_{4,t} = \text{forecasts at leads 0 through 4}.\]

In the main SPF experiment, the $x_{k,t}$ are not individual named forecasters. They are the panel consensus forecasts at different lead times: nowcast, one-quarter-ahead, …, four-quarters-ahead. I will use the word “signal” for any predictor being combined: a lead-time consensus, an individual forecaster, or a model output.

There are two natural ways to use copulas here.

Approach 1: outcome-forecast dependence. Model the joint dependence of

\[(y_t, x_{0,t}, \ldots, x_{4,t})\]

after rank transformation. The question is:

\[\mathbb{E}[y_t \mid x_{0,t}, \ldots, x_{4,t}]?\]

In rank space this becomes: given the forecast ranks, what rank should the realised outcome have? The Gaussian copula version reduces to OLS, or best linear prediction, on probit-transformed ranks. A vine copula can replace the Gaussian copula when the target-forecast relationship is nonlinear, asymmetric, or tail-dependent.

Approach 2: forecast-error dependence. Model the joint dependence of forecast errors

\[e_{k,t} = x_{k,t} - y_t.\]

The question is different:

\[\hat y_t = \arg\max_y f_E(x_{0,t} - y, \ldots, x_{4,t} - y).\]

In words: choose the combined forecast whose implied error vector is most plausible under the learned joint error distribution. This is the direct continuation of my older copula-fusion intuition: learn how predictors err together, then find the value that makes the observed forecasts jointly most likely.

This post is about point forecast ranks and point forecast errors, not full predictive densities. Copulas are often used to combine density forecasts; here they play a narrower role. They model dependence either between outcome and forecasts, or among forecast errors.

Classical forecast combination

Bates and Granger (1969) showed that a weighted average of forecasts can outperform the individual forecasts. For two forecasts with error variances $\sigma_1^2$, $\sigma_2^2$ and error correlation $\rho$, the optimal weight on forecast 1 is

\[w_1^* = \frac{\sigma_2^2 - \rho\sigma_1\sigma_2}{\sigma_1^2 + \sigma_2^2 - 2\rho\sigma_1\sigma_2}.\]

The formula already contains the main idea: a good combination needs both individual accuracy and error dependence. If two forecasters make the same mistakes, the second one does not add a full new unit of information.

The practical problem is estimation. Variances and covariances must be learned from finite history. When there are many forecasters, the covariance matrix has many parameters, and the inverse can amplify noise. This is one way the forecast combination puzzle appears: estimated optimal weights often fail to beat the simple average out of sample.

Granger and Ramanathan (1984) proposed regression-based forecast combination,

\[y_t = \alpha + \sum_k w_k x_{k,t} + \varepsilon_t,\]

which is flexible but has the same small-sample fragility. A simpler estimated baseline is inverse MSE weighting: estimate each forecast’s recent mean squared error, set $w_k \propto 1/\mathrm{MSE}_k$, and renormalise the weights to sum to one. It detects persistent skill differences without estimating the full error covariance matrix. Genre et al. (2013), in a detailed SPF study, found that simple averages and trimmed means were hard to beat for same-horizon macro forecast combination.

That is the background tension for the rest of this post. We want to use dependence, but we do not want to overfit it.

Ranks, pseudo-observations, and probits

A copula separates marginal distributions from dependence. The standard first step is to turn each variable into a pseudo-observation:

\[u_i = \frac{\#\{j : x_j \leq x_i\}}{n+1}.\]

The $+1$ keeps values away from exactly 0 and 1. After this transform, every variable is approximately uniform on $(0,1)$.

For the Gaussian copula we then apply the probit transform

\[z_i = \Phi^{-1}(u_i),\]

where $\Phi$ is the standard normal CDF. A median rank $u=0.50$ maps to $z=0$; $u=0.84$ maps to about $z=1$; $u=0.025$ maps to about $z=-1.96$.

The probit transform preserves the ordering of each individual variable, but it changes the geometry in which linear combinations are formed. A weighted sum of probit ranks need not rank observations exactly like a weighted sum of raw ranks. In this series, the probit transform is a modelling choice: it puts the rank data into the normal-score space where Gaussian conditioning has a closed form.

Before introducing the error-side approach, it is useful to keep two rank-side methods separate:

Name in this series	Data being combined	Estimated object	Main intuition
OLS on raw ranks	$u_y$ on $u_1,\ldots,u_D$	regression weights in rank space	best linear rank predictor without a Gaussian-copula assumption
Gaussian outcome copula	$z_y=\Phi^{-1}(u_y)$ on $z_1,\ldots,z_D$	best linear predictor in normal-score space	same regression idea, but after the probit map that makes Gaussian conditioning closed form

Both rank-side methods are primarily rank predictors. They are good for ordering outcomes, comparing forecasters, or measuring Spearman correlation. They are not automatically calibrated level forecasts. To recover a level forecast, one has to map the predicted rank back through an estimated marginal distribution of realised outcomes:

\[\hat y = \hat F_Y^{-1}(\hat u_y),\]

where $\hat F_Y$ is fitted on the training sample, usually separately by variable. This works as a monotone calibration step, but it inherits the limits of the historical distribution: it struggles to extrapolate beyond past realised values and may be poorly calibrated after regime shifts. That is why the empirical sections mostly evaluate rank skill. Raw-error methods, such as the classical Gaussian GLS average, work directly in the original forecast units. Error-copula methods also search over candidate outcome levels $y$: for each candidate, they compute implied level errors $e_k(y)=x_k-y$, transform those errors through the fitted marginal error distributions, and score the resulting error-rank vector under the fitted copula.

Approach 1: outcome-forecast dependence

Let $u_y$ be the realised outcome rank and let $\mathbf{u}_x = (u_1, \ldots, u_D)$ be the forecast ranks. A copula model for $(u_y, \mathbf{u}_x)$ induces a conditional distribution of the outcome rank given the forecast ranks, which I will denote informally by

\[C(u_y \mid u_1, \ldots, u_D).\]

A natural rank forecast is a summary of that conditional distribution, such as its mean or median. With a Gaussian copula, the clean closed-form object is the conditional mean in probit space. Define

\[z_y = \Phi^{-1}(u_y), \qquad \mathbf{z}_x = \Phi^{-1}(\mathbf{u}_x).\]

Assume

\[\begin{pmatrix} z_y \\ \mathbf{z}_x \end{pmatrix} \sim \mathcal{N}\left( 0, \begin{pmatrix} 1 & \rho^\top \\ \rho & \Sigma_{xx} \end{pmatrix} \right).\]

Then the conditional mean in normal-score space is

\[\mathbb{E}[z_y \mid \mathbf{z}_x] = \rho^\top \Sigma_{xx}^{-1}\mathbf{z}_x.\]

Equivalently,

\[\hat z_y = \mathbf{w}^\top \mathbf{z}_x, \qquad \mathbf{w} = \Sigma_{xx}^{-1}\rho.\]

This is the population OLS coefficient from regressing $z_y$ on $\mathbf{z}_x$:

\[\mathbf{w}_{OLS} = \mathbb{E}[\mathbf{z}_x\mathbf{z}_x^\top]^{-1}\mathbb{E}[\mathbf{z}_x z_y] = \Sigma_{xx}^{-1}\rho.\]

The clean name is: OLS on probit-transformed ranks.

After computing $\hat z_y$, I map back with $\hat u_y=\Phi(\hat z_y)$. This is the conditional median of $u_y$ under the Gaussian copula (the exact conditional mean is $\Phi(m/\sqrt{1+s^2})$, with $m=\rho^\top\Sigma_{xx}^{-1}z_x=\hat z_y$ and $s^2=1-\rho^\top\Sigma_{xx}^{-1}\rho$, but this distinction does not affect rank evaluation, since both quantities are monotone functions of the same conditional Gaussian location; it matters only if the output is interpreted as a calibrated conditional rank).

The key object is $\Sigma_{xx}^{-1}$: it downweights redundant signals. Suppose two forecasts are almost identical and a third forecast is independent. If all three are equally predictive of the target, $\Sigma_{xx}^{-1}$ gives the two redundant forecasts roughly half weight each, while the independent forecast keeps its full weight. Equal-weighting misses this distinction.

Vine copulas for outcome-forecast dependence

The Gaussian copula is restrictive: dependence is symmetric and linear in probit space. A vine copula replaces this with a product of bivariate pair copulas. With a C-vine rooted at the target,

\[p(u_y, u_1,\ldots,u_D) \approx p(u_y)\prod_{k=1}^D c_k(u_y,u_k).\]

The conditional mean becomes

\[\mathbb{E}[u_y \mid u_1,\ldots,u_D] = \frac{\int_0^1 u \prod_k c_k(u,u_k)\,du} {\int_0^1 \prod_k c_k(u,u_k)\,du}.\]

This one-dimensional integral can be evaluated by Gauss-Legendre quadrature. Each pair copula can be Gaussian, Student-t, Gumbel, Clayton, Frank, or a rotated version, selected by BIC. Here, a rotated version means a reflection of an asymmetric copula, useful for opposite-tail or negative dependence patterns.

Non-Gaussian pair copulas can help when a target-forecast relationship is not well described by a Gaussian copula. The figure below fixes Kendall $\tau$ at 0.5 for several bivariate copula families. They have the same overall rank association, but they transform the same signal rank into different expected outcome ranks, especially in the tails. A vine uses these pair copulas as building blocks inside a multivariate dependence model. The dotted diagonal is only an identity benchmark, $E[u_y\mid u_s]=u_s$: it maps a signal rank directly to the same outcome rank.

But the vine used here is deliberately simplified. A full vine over $(u_y,u_1,\ldots,u_D)$ would also model the dependence among the forecasts themselves, including conditional relationships such as forecast 1 versus forecast 2 after accounting for the target. That is more complete, but it requires many more pair-copulas and much more data.

The truncated target-rooted vine keeps only the target-signal links $c_k(u_y,u_k)$. In plain language, it asks: “how does each forecast relate to the realised outcome?” and then multiplies those pieces together. That multiplication quietly assumes that, once the realised outcome is known, the forecast signals are independent of each other. This is often false for SPF lead-time forecasts. If the nowcast and the one-quarter-ahead forecast both reflect the same public macro release, the truncated vine can count that same evidence twice. The Gaussian inverse $\Sigma_{xx}^{-1}$ corrects for redundancy directly; this truncated vine does not. This is not a limitation of vines in general: a fuller vine could add conditional signal-signal pair copulas and model that redundancy. The cost is statistical, because those extra pair copulas consume data. The trade-off here is therefore specific to the truncated approximation: it is attractive for nonlinear target-signal shapes when the signals themselves are not too redundant; otherwise the Gaussian copula’s analytical inter-signal correction is often more reliable in small macro samples.

Approach 2: forecast-error dependence

The second approach models errors directly. Write

\[X_k = Y + E_k.\]

If the forecast-error vector is Gaussian and centred,

\[E = (E_1,\ldots,E_D)^\top \sim \mathcal{N}(0,\Sigma_E),\]

then, for a candidate value $y$, the implied error vector is

\[e(y) = x - y\mathbf{1}.\]

The log likelihood is, up to constants,

\[-\frac12 (x-y\mathbf{1})^\top \Sigma_E^{-1}(x-y\mathbf{1}).\]

Maximising this over $y$ gives

\[\hat y = \frac{\mathbf{1}^\top \Sigma_E^{-1}x} {\mathbf{1}^\top \Sigma_E^{-1}\mathbf{1}}.\]

This is the precision-weighted forecast combination.

Why call it Error GLS? GLS stands for generalized least squares. The analogy refers specifically to the inverse covariance of forecast errors. This is different from the outcome-side Gaussian copula above: there, $\Sigma_{xx}^{-1}$ is the inverse of the signal covariance matrix in probit-rank space. Here, $\Sigma_E^{-1}$ is the inverse of the forecast error covariance matrix in level space. Both inverses change the geometry of the combination problem so that redundant or noisy directions count less.

The covariance matrix is not the residual covariance from a regression of $y$ on $x$. It is the covariance of the forecasters’ errors with each other:

\[\Sigma_E = \operatorname{Cov}(E_1,\ldots,E_D).\]

So Error GLS means: estimate how the forecasts err together, then combine the observed forecasts with weights implied by $\Sigma_E^{-1}$. If two forecasts usually make the same mistake, their shared error mode receives less weight. If one forecast has low error variance after accounting for the others, it receives more weight.

This is the approach that most deserves the GLS analogy because it is explicitly using the inverse covariance matrix of forecast errors. The zero-mean assumption is only for the clean derivation. In practice, one should either de-bias the forecast errors on the training sample or include an estimated mean error vector $\mu_E$; otherwise the method may combine forecasts precisely around the wrong centre.

It is also useful to compare the two inverse matrices that have appeared:

Method	Inverse matrix	Space	What gets discounted?
Gaussian outcome copula	$\Sigma_{xx}^{-1}$	probit-transformed forecast ranks	redundant signals
Error GLS	$\Sigma_E^{-1}$	forecast errors in level space	redundant error modes

The formulas look similar because both use a precision matrix, but they answer different probability questions. The outcome-side approach asks for the likely rank of $Y$ given the forecast ranks. The error-side approach asks which candidate $y$ makes the implied forecast errors most plausible.

The true copula-error generalisation

The Gaussian error model can be generalised with a copula. Let each error have marginal CDF $F_k$ and density $f_k$. Then

\[f_E(e_1,\ldots,e_D) = c(F_1(e_1),\ldots,F_D(e_D))\prod_k f_k(e_k).\]

For each candidate $y$, compute implied errors

\[e_k(y)=x_k-y.\]

Then choose

\[\hat y = \arg\max_y \left[ \log c(F_1(e_1(y)),\ldots,F_D(e_D(y))) + \sum_k \log f_k(e_k(y)) \right].\]

This is the direct copula version of the “most plausible errors” idea. I estimate the marginal error distributions with KDE, transform candidate errors to error ranks, and fit either a Gaussian copula or a vine copula to those transformed errors. Prediction then becomes a one-dimensional optimisation over candidate $y$ values: choose the value whose implied transformed error vector has the highest fitted copula likelihood. In the experiments I use a robust grid search rather than a gradient method, because the KDE-plus-copula likelihood can be irregular in small samples.

This method is conceptually appealing, but it is more fragile than the closed-form Gaussian formula. It requires good marginal error estimates, enough data to fit the copula, and a stable optimisation for every prediction.

Synthetic checks

The synthetic studies below check whether the methods behave as theory says they should. For the outcome-side approach, Case A is a synthetic data-generating process with correlated Gaussian signals, where redundancy correction should help; Case B is a synthetic data-generating process with tail-dependent signals, where a vine has a real nonlinear dependence pattern to learn. For the error-side approach, I use one synthetic design with Gaussian correlated forecast errors and one with curved forecast-error dependence.

For the cross-sectional synthetic experiments, I report ICIR, the information coefficient ratio: the mean rank correlation between prediction and realised outcome, divided by its time-series standard deviation. The synthetic setup generates $T$ periods, each with $N$ cross-sectional observations; the information coefficient (IC) is the Spearman rank correlation between predicted and realised outcomes within each period. ICIR is then the mean IC divided by its standard deviation across the $T$ periods. Higher ICIR means the method is not only predictive on average, but predictably predictive through time.

Outcome-side approach: when Gaussian helps and when vine helps

In the outcome-forecast synthetic study, Case A has correlated Gaussian signals. The Gaussian copula edges out equal-weight because $\Sigma_{xx}^{-1}$ deflates redundant signal clusters:

Method	Case A ICIR
Equal-weight	9.1
Gaussian copula	9.2
Vine copula	8.8

Case B has independent signals with non-Gaussian target-signal dependence: Gumbel upper-tail, survival Gumbel lower-tail, Student-t heavy tails, and Gaussian. The vine recovers the pair-copula families and wins:

Method	Case B ICIR
Equal-weight	97.5
Gaussian copula	124.4
Vine copula	129.1

The detailed synthetic comparison is shown below.

The synthetic example illustrates the intended role of the vine: it can improve on the Gaussian copula when the target-signal dependence is genuinely asymmetric, tail-dependent, or otherwise non-elliptical. That extra flexibility is useful only if the sample contains enough stable dependence signal to estimate it reliably. The empirical question is whether the SPF panel provides that much information.

Error-side approach: when GLS wins and when vine wins

In the error-dependence study, the Gaussian correlated-error case behaves as expected:

Method	Spearman	RMSE
Equal-weight	0.853	0.609
Inverse MSE	0.888	0.513
Outcome Gaussian rank	0.921	0.398
Error GLS	0.928	0.402
Gaussian error-copula	0.925	0.391
Vine error-copula	0.893	0.510

The RMSE for Outcome Gaussian rank is computed after mapping its predicted rank back through the empirical training distribution of the realised outcome. In this Gaussian-error design, the error covariance is the right summary of the error distribution. Error GLS therefore gives the best rank correlation, while the Gaussian error-copula gives the best level RMSE. The vine error-copula is more flexible, but that flexibility is unnecessary here and hurts out-of-sample performance.

But we can also construct a nonlinear error geometry where a covariance matrix is the wrong summary. Let a latent error factor $a$ drive

\[E_1 \approx a, \qquad E_2 \approx a^2 - 1, \qquad E_3 \approx -(a^2 - 1).\]

The dependence is curved, not elliptical.

In that case the vine error-copula wins:

Method	Spearman	RMSE
Equal-weight	0.972	0.248
Inverse MSE	0.959	0.307
Error GLS	0.995	0.089
Outcome Gaussian rank	0.989	0.157
Gaussian error-copula	0.979	0.198
Vine error-copula	0.995	0.079

The RMSE for Outcome Gaussian rank is again computed after empirical rank-to-level calibration. It beats the Gaussian error-copula here because it directly models the target rank $Y \mid X_1,\ldots,X_D$, while the Gaussian error-copula is an indirect error-likelihood model and is misspecified for this curved error distribution. Error GLS also remains strong: although the dependence is nonlinear, $E_2$ and $E_3$ nearly cancel each other, so the covariance matrix still captures a powerful linear variance-reduction effect. The vine error-copula wins by using the remaining nonlinear error geometry that covariance compresses into an ellipse.

A humbling SPF fixed-split check

Synthetic data is useful because the data-generating process is known. The SPF panel is a harder test: the signal-to-noise ratio, the dependence structure, and the stability of the relationships are all uncertain.

Before the full rolling experiment of Part 3, I ran a simple fixed split on the SPF panel: train through 2010, test after 2010, requiring all five lead-time forecasts to be present. Each method is fitted separately by macro variable so that GDP growth, inflation, unemployment, housing, and industrial production are not forced onto one common scale. Rank/probit transforms, error KDEs, and covariance matrices are estimated on the training sample only. For evaluation, realised values and predictions are first converted to ranks within each macro variable; those within-variable ranks are then stacked, and one Spearman correlation is computed on the stacked rank data.

Method	Spearman on stacked within-variable ranks
Outcome Gaussian rank	0.511
Error GLS	0.478
Inverse MSE	0.436
Equal-weight	0.406
Gaussian error-copula	0.322
Vine error-copula	0.296

This first real-data check is humbling. The KDE plus copula-error likelihood underperforms here, which is consistent with a method that is too flexible for the available macro sample and for the stability of the train-test relationship. The outcome Gaussian rank model estimates one main object: a signal correlation matrix fitted to rank-transformed data. The error-copula approach estimates several objects in sequence: a KDE for each error marginal, then a multivariate copula fitted on the transformed error ranks. With only a few decades of quarterly observations, this two-stage estimation problem can add more variance than useful structure. Error GLS is less expressive, but it is also much lighter: it collapses the error dependence into a single parametric covariance matrix, which is why it holds up better in this check.

This is a useful caution for macroeconomic and financial forecasting: a flexible model can win in a synthetic design where its assumptions are true, but lose in real samples unless it is strongly regularised.

What Part 3 tests

Part 3 compares these method families on a walk-forward SPF horserace:

Family	Method	What it tests
Baselines	Persistence, Lead 0, equal-weight, trimmed mean, inverse MSE	How far simple benchmarks go
Outcome-side approach	OLS on ranks, Gaussian copula on probit ranks, vine outcome copula	Whether modelling target-forecast dependence helps
Error-side approach	Error GLS, shrunk error GLS, Gaussian/vine error-copula	Whether modelling forecast-error dependence helps

Part 3 will use a walk-forward evaluation with an expanding training window: the test date moves forward one forecast date at a time, and each fitted method is refitted using only the history available at that date. This is closer to actual forecast use than the fixed split above. It will also compare initial-release and latest-revised targets, and trace rolling out-of-sample performance through time. The named-forecaster experiment will remain separate, because same-horizon forecasters with similar skill are a different problem from cross-horizon signals with a structural lead-time hierarchy.

The key distinction is between two probability questions:

\[\mathbb{E}[Y \mid X_1,\ldots,X_D]\]

versus

\[\arg\max_y f_E(X_1-y,\ldots,X_D-y).\]

The first models the outcome conditional on the forecasts. The second models the joint forecast errors implied by a candidate outcome.

References and further reading

Bates, J. M. and Granger, C. W. J. (1969), “The Combination of Forecasts”, Operational Research Quarterly.
Granger, C. W. J. and Ramanathan, R. (1984), “Improved Methods of Combining Forecasts”, Journal of Forecasting.
Genre, V., Kenny, G., Meyler, A. and Timmermann, A. (2013), “Combining Expert Forecasts: Can Anything Beat the Simple Average?”, International Journal of Forecasting.
Sklar, A. (1959), “Fonctions de repartition a n dimensions et leurs marges,” Publications de l’Institut de Statistique de l’Universite de Paris.
Bedford, T. and Cooke, R. M. (2002), “Vines: A New Graphical Model for Dependent Random Variables”, Annals of Statistics.
Aas, K., Czado, C., Frigessi, A. and Bakken, H. (2009), “Pair-Copula Constructions of Multiple Dependence”, Insurance: Mathematics and Economics.

Combining Professional Forecasters with Copulas (Part 1): The Survey of Professional Forecasters

Sat, 04 Jul 2026 00:00:00 +0000

This is the first post in a three-part series. Part 1 introduces the Survey of Professional Forecasters and provides a thorough exploratory analysis of the dataset. Part 2 develops the copula framework and validates it on synthetic data. Part 3 runs the walk-forward empirical horserace on official SPF evaluation targets.

Why forecast combination matters

You have several forecasters. Each one issues quarterly predictions for an economic variable, say the unemployment rate two quarters from now. None of them is reliably the best. They agree with each other more than they disagree. How should you combine them into a single estimate that is better than any individual forecaster and, ideally, better than simply averaging all of them?

This question has occupied economists since Bates and Granger’s 1969 paper introduced the combination framework. It has a subtlety that is easy to miss: the naive answer (“optimally weight them”) often fails in practice. The optimal weights, estimated from data, carry enough estimation noise that a simple equal-weight average frequently beats them out-of-sample. This empirical regularity is called the forecast combination puzzle.

Over this three-part series we will use a copula-based combination method to probe when and why the puzzle appears and disappears. The dataset is the Survey of Professional Forecasters (SPF): 58 years of quarterly macroeconomic predictions, freely available from the Federal Reserve Bank of Philadelphia. This post is entirely about understanding that dataset before fitting any model.

The Survey of Professional Forecasters: a brief history

The SPF was launched in 1968, making it one of the oldest continuously-run macroeconomic forecast surveys in the world. It was originally co-sponsored by the American Statistical Association (ASA) and the National Bureau of Economic Research (NBER). In 1990, the Federal Reserve Bank of Philadelphia took over administration and has run the survey ever since without interruption.

The survey is conducted quarterly, near the middle of each quarter, before the Bureau of Economic Analysis releases its advance GDP estimate. Participants are professional economists at banks, investment firms, research institutions, and universities. Participation is voluntary and anonymous: each respondent is identified by a numeric ID that appears consistently across surveys but is never linked to a name or institution in the public data release.

Why central banks pay attention. The SPF is a primary external cross-check for the Federal Reserve’s own internal Greenbook projections. In the period before the Greenbook became public, the SPF was one of the few sources of professional near-term forecasts that the Fed could compare itself against. Today, the Fed’s Monetary Policy Report cites SPF long-run inflation expectations as a measure of inflation anchoring. The European Central Bank and the Bank of England run structurally similar surveys (the ECB SPF and the Bank of England’s Monetary Policy Committee surveys), partly inspired by the U.S. SPF.

Why the academic literature uses it. The combination of individual-level microdata, long history, and multiple variables makes the SPF unusually rich for econometric research. Key papers that use it include:

Ang, Bekaert, and Wei (2007): showed that the SPF consensus outperforms a wide range of model-based inflation forecasts, including Phillips-curve, term-structure, and time-series models.
Coibion and Gorodnichenko (2015): used the individual-level microdata to document information rigidity, i.e., professional forecasters underreact to publicly available data in a way consistent with sticky information models.
Genre, Kenny, Meyler, and Timmermann (2013): studied whether any combination method beats the simple average for same-horizon SPF forecasts and found the puzzle alive and well.

The dataset: variables, horizons, and structure

The SPF collects predictions for a set of U.S. macroeconomic variables at five horizons each quarter: the nowcast (the current quarter, which may be partially observed), and four-quarters-ahead. The eight variables used in this series are described below.

Real GDP growth (RGDP)

What it measures. Real Gross Domestic Product is the total market value of all goods and services produced in the United States in a given quarter, adjusted for inflation using chain-weighted price indices with a base year of 2017. It is the broadest single measure of economic activity.

How it is collected. The Bureau of Economic Analysis (BEA) releases three successive vintages each quarter: the advance estimate (about four weeks after quarter-end), the second estimate (about eight weeks), and the third estimate (about twelve weeks). SPF forecasters submit responses near the middle of the quarter, so the advance estimate for the just-completed quarter is not yet available; forecasters are largely working from monthly indicators (employment, retail sales, industrial production).

Units in SPF. Forecasters report in billions of chained 2017 dollars at a seasonally adjusted annual rate. We convert to quarter-on-quarter growth rates (%) using the ratio of consecutive horizon values from the same respondent’s row.

Why it matters. Real GDP growth is the headline indicator for business cycle expansions and recessions, the variable the Federal Open Market Committee implicitly or explicitly targets through the dual mandate, and the reference series for most macroeconomic models. Predicting it correctly is notoriously difficult: GDP growth is close to a random walk over quarterly horizons once trend growth is removed.

GDP price deflator (PGDP)

What it measures. The GDP price deflator (also called the GDP implicit price deflator) is the ratio of nominal GDP to real GDP; it covers the prices of all goods and services included in GDP, including investment, government spending, and net exports. It is therefore broader than the CPI, which covers only household consumption.

How it is collected. Published by the BEA alongside GDP releases. Subject to the same vintage structure as RGDP.

Units in SPF. Forecasters report index levels (2017 = 100). We convert to quarter-on-quarter growth rates.

Why it matters. The GDP deflator is the broadest measure of domestic price pressure. The Federal Reserve’s preferred inflation gauge is the PCE deflator (Personal Consumption Expenditures), which is closely related but covers a slightly different basket. Academic forecasting studies often use PGDP because its history in the SPF is longer than the PCE series.

Nominal GDP (NGDP)

What it measures. Nominal GDP is GDP measured at current market prices, without any inflation adjustment. By accounting identity, NGDP growth equals real GDP growth plus the GDP deflator inflation rate. It is therefore the sum of the two previous variables.

Units in SPF. Billions of current (nominal) dollars at a seasonally adjusted annual rate, converted to quarter-on-quarter growth rates.

Why it matters. Nominal GDP targeting was proposed as an alternative monetary policy framework (McCallum 1987, Sumner 2014): the idea is that a central bank targeting a stable NGDP growth path would automatically accommodate supply shocks without generating unnecessary recessions. The variable thus has a direct policy relevance beyond its role as a bookkeeping identity.

Industrial production (INDPROD)

What it measures. The Federal Reserve’s Industrial Production index tracks the real output of the manufacturing, mining, and electric and gas utilities sectors. It covers roughly one fifth of the U.S. economy but is more cyclically sensitive than GDP because it excludes the service sector, which is less volatile.

How it is collected. Produced by the Federal Reserve Board and released monthly, giving forecasters more within-quarter information than for GDP.

Units in SPF. Index levels (2017 = 100), converted to quarter-on-quarter growth rates.

Why it matters. Because industrial production is available monthly, it serves as a key input to GDP nowcasting models. The Chicago Fed National Activity Index and the Philadelphia Fed’s ADS index both weight industrial production heavily. Its higher frequency makes it more predictable at short horizons from within-quarter data, but it is more volatile than GDP at longer horizons.

Housing starts (HOUSING)

What it measures. Housing starts counts the number of new privately owned residential units on which construction has begun in a given month. The SPF variable is a quarterly aggregate.

How it is collected. Published monthly by the U.S. Census Bureau and the Department of Housing and Urban Development, available with only a short lag.

Units in SPF. Thousands of units at a seasonally adjusted annual rate, converted to quarter-on-quarter growth rates.

Why it matters. Housing starts is one of the most reliable leading indicators of the business cycle: residential investment typically peaks well before recessions and troughs early in recoveries. The series is highly sensitive to mortgage rates, which means it reflects Federal Reserve policy with a lag of six to eighteen months. As a result it is watched closely by financial market participants. However, it is also highly volatile and subject to weather effects, making medium-term forecasting particularly hard.

Corporate profits (CPROF)

What it measures. The BEA’s after-tax corporate profits series measures the operating surplus earned by domestic corporations, adjusted for inventory valuation and capital consumption. It is a quarterly flow variable.

How it is collected. Published by the BEA as part of the National Income and Product Accounts (NIPA), with substantial revision risk: the data is subject to benchmark revisions that can be large and retrospective.

Units in SPF. Billions of current dollars, converted to quarter-on-quarter growth rates.

Why it matters. Corporate profits are a leading indicator of future investment and hiring decisions: firms that earn more tend to invest more. They are also directly relevant to equity valuation (equity prices are discounted future profits), and they are an important input to fiscal projections because corporate tax revenues track profits with a short lag. Despite their economic importance, corporate profits are one of the hardest SPF variables to forecast, partly because of their volatility and partly because of data revision uncertainty.

Unemployment rate (UNEMP)

What it measures. The unemployment rate is the share of the civilian labour force that is without a job, actively seeking work, and available to work. It is the most widely followed labour market indicator.

How it is collected. The Bureau of Labor Statistics (BLS) computes the rate from the monthly Current Population Survey (CPS), a household survey of approximately 60,000 households. It is released on the first Friday of the following month and is not subject to large revisions (unlike GDP).

Units in SPF. Percentage of the labour force, reported directly as a level (not a growth rate). This is important: because unemployment is a rate, it is already stationary and mean-reverting, and forecasters submit it as a level rather than a change.

Why it matters. Unemployment is one of the two variables in the Federal Reserve’s dual mandate (maximum employment and price stability). It is also a key input to wage dynamics and therefore to inflation (the Phillips curve links unemployment gaps to inflation). Its serial persistence is extremely high: the first-order autocorrelation of quarterly unemployment is above 0.97 over the full SPF sample. This means that even a naive “next quarter will look like this quarter” forecast achieves Spearman rank correlation above 0.95 with the realised outcome, a benchmark that combination methods must beat.

CPI inflation (CPI)

What it measures. The Consumer Price Index for All Urban Consumers (CPI-U) measures the average change over time in the prices paid by urban consumers for a representative basket of goods and services. The basket includes food, energy, housing, medical care, transportation, and recreation, weighted by expenditure shares from the Consumer Expenditure Survey.

How it is collected. Each month, BLS field agents physically record prices for roughly 80,000 items across approximately 23,000 retail outlets, service providers, and rental units (grocery stores, hospitals, landlords, gas stations, etc.) in 75 urban areas. Items are organised into a fixed consumption basket whose weights come from the Consumer Expenditure Survey, which tracks what households actually spend. Housing costs, the single largest component, are measured via owners’ equivalent rent, estimated by surveying renters about what their units would fetch on the open market. The BLS releases the resulting index monthly, typically in the second or third week of the following month. Unlike GDP, CPI is not subject to major revisions after initial release; only seasonal adjustment factors are updated annually, and those changes are modest.

Units in SPF. The SPF asks for the percentage change in the CPI over the relevant forecast horizon, expressed at an annual rate. This is one of the rate variables reported directly rather than as a level.

Why it matters. CPI is the most visible inflation measure for households, wage negotiators, and indexed contracts (Social Security benefits, TIPS coupons, many labour agreements are CPI-indexed). The Fed targets a 2% rate of PCE inflation, which historically runs about 0.3-0.5 percentage points below CPI, but the public and markets watch CPI closely. The unpredictability of CPI beyond very short horizons (the nowcast captures quarter-to-quarter momentum, but by h=2 the correlation with realised values drops to around 0.32 for the consensus and near zero for individual forecasters) reflects genuine macroeconomic uncertainty: inflation depends on energy prices, supply chains, and expectations, none of which are easily projected two quarters out.

A compact summary:

Code	What is forecasted	Units	Level or rate
RGDP	Real GDP	q/q growth %	Level → growth
PGDP	GDP price deflator	q/q growth %	Level → growth
NGDP	Nominal GDP	q/q growth %	Level → growth
INDPROD	Industrial production index	q/q growth %	Level → growth
HOUSING	Housing starts	q/q growth %	Level → growth
CPROF	Corporate profits	q/q growth %	Level → growth
UNEMP	Unemployment rate	% of labour force	Rate (direct)
CPI	CPI-U inflation	% annual rate	Rate (direct)

Level variables are reported by forecasters as quarterly index levels; we convert to quarter-on-quarter growth rates using the ratio of consecutive horizon columns. Rate variables are reported directly as percentages and require no transformation.

The full microdata file is publicly available at no charge from the Philadelphia Fed:

import pandas as pd

# Download once from Philadelphia Fed:
# https://www.philadelphiafed.org/surveys-and-data/real-time-data-research/individual-forecasts
# Then read with calamine engine (openpyxl fails on this file's date types)

xl = pd.ExcelFile("spfmicrodata.xlsx", engine="calamine")
print(xl.sheet_names)
# ['NGDP', 'PGDP', 'CPROF', 'UNEMP', 'EMP', 'INDPROD', 'HOUSING',
#  'TBILL', 'BOND', 'BAABOND', 'TBOND', 'RGDP']

Each sheet has the structure:

YEAR  QUARTER  ID   RGDP1  RGDP2  RGDP3  RGDP4  RGDP5  RGDP6  ...
1968  4        1    713.9  720.1  730.2  ...
1968  4        2    713.9  735.0  748.0  ...

The column convention is: {VAR}1 is the previous quarter’s level (provided as a common anchor), {VAR}2 is the nowcast (h=0), {VAR}3 is h=1, through {VAR}6 which is h=4.

Building realised values. We reconstruct realised outcomes from the survey data itself: for target quarter $t$, the realised value is the median of the panel’s h=0 (nowcast) column in the survey for quarter $t+1$. By the time the next survey is conducted, the BEA advance estimate is already public, so the following quarter’s nowcast is effectively a read on the just-completed quarter. This is a standard real-time proxy used in the forecast evaluation literature (Croushore and Stark 2001). It is important to acknowledge that this is not the same as the final revised BEA estimate and may differ materially for periods with large revisions, notably around recessions and 2020-2021.

def qadd(y, q, k):
    """Advance (y, q) by k quarters."""
    idx = y * 4 + (q - 1) + k
    return idx // 4, idx % 4 + 1

# For a level variable like RGDP:
# realised growth for quarter t = median(RGDP2) in survey t+1
# where RGDP2 is the h=0 (nowcast) column
actuals = {}
for (y, q), grp in df.groupby(["YEAR", "QUARTER"]):
    median_nowcast = grp["RGDP2"].median()
    if pd.notna(median_nowcast):
        ty, tq = qadd(y, q, -1)        # this nowcast refers to the prior quarter
        actuals[(ty, tq)] = median_nowcast

# Convert levels to growth rates
realized_growth = {}
for (y, q), level in sorted(actuals.items()):
    ty, tq = qadd(y, q, -1)
    if (ty, tq) in actuals:
        realized_growth[(y, q)] = level / actuals[(ty, tq)] - 1

EDA 1: panel size and forecaster turnover

The first thing to understand about the SPF is how the panel has evolved over 58 years.

The panel grew rapidly in the 1970s, contracted in the late 1980s, then stabilised after the Philadelphia Fed took over administration in 1990. Since 1990, the RGDP panel has consistently attracted 30-50 respondents per quarter, with a brief contraction during the 2008-2009 financial crisis and a more sustained one after 2020 as some long-tenured participants retired.

The turnover problem. Of the 463 unique forecaster IDs in the RGDP file, most are active for a short stretch:

The median forecaster submits RGDP predictions for only about 9 quarters. The 90th percentile is around 54 quarters (13.5 years). Even the most persistent forecaster in the RGDP panel appears in only 126 of 231 surveys (54%). This has an important modelling consequence: a balanced panel of named forecasters spanning the full 58-year history is not feasible. We address this in two ways in the later posts: by using the panel consensus (median) across all active respondents at each horizon, and by running a named-forecaster experiment restricted to the sub-period where at least three of the most persistent respondents are simultaneously active.

import numpy as np

# Load RGDP sheet
df = pd.read_excel(xl, sheet_name="RGDP", engine="calamine")

# Count quarters per forecaster
tenure = df.groupby("ID")["YEAR"].count()

print(f"Unique IDs: {df['ID'].nunique()}")
print(f"Median tenure (quarters): {tenure.median():.0f}")
print(f"90th percentile: {tenure.quantile(0.9):.0f}")
print(f"Max (most persistent): {tenure.max()}")
print(f"Fraction of total 231 quarters: {tenure.max() / 231:.1%}")
# Unique IDs: 463
# Median tenure (quarters): 9
# 90th percentile: 54
# Max (most persistent): 126
# Fraction of total 231 quarters: 54.5%

EDA 2: how predictable are these variables?

Before thinking about combination, the most basic question is whether professional forecasters have any skill at all, and how that skill varies across variables and horizons.

The pattern is striking and variable-specific:

Unemployment rate is the most forecastable variable at every horizon. Even four quarters ahead, the consensus has a Spearman rank correlation with the realised rate above 0.30. The nowcast correlation exceeds 0.99. This reflects the strong serial persistence of unemployment: the level this quarter predicts next quarter’s level with very high accuracy.
CPI inflation forecast skill decays quickly. The nowcast correlation is around 0.60 (the BLS releases CPI for the first month of the quarter before the survey date, giving forecasters a partial read on current-quarter inflation), but by h=2 the consensus correlation drops to around 0.32, and individual forecaster skill is near zero. The consensus still extracts a weak common signal, but individual precision is essentially gone.
Real GDP growth is difficult to forecast even one quarter ahead. The nowcast correlation (around 0.55) reflects the within-quarter monthly indicators already available at survey time (payrolls, retail sales, industrial production) rather than the BEA advance GDP estimate, which is not yet published when forecasters submit; the one-quarter-ahead forecast already struggles to beat persistence.

The fact that predictability varies so much across variables is important for the combination design: any method that pools across variables must account for this heterogeneity.

from scipy.stats import spearmanr

def consensus_skill_by_horizon(var, panel_df, realized_df):
    """Compute Spearman ρ between consensus forecast and realised, for each horizon."""
    rhos = []
    for h in range(5):
        col = f"{var}{h + 2}"   # h=0 -> col 2, h=4 -> col 6
        if col not in panel_df.columns:
            rhos.append(np.nan); continue
        # Panel consensus = median across all respondents in each survey quarter
        consensus = panel_df.groupby(["YEAR", "QUARTER"])[col].median().reset_index()
        # Advance survey date to the target quarter
        consensus["target_y"] = consensus.apply(
            lambda r: qadd(int(r.YEAR), int(r.QUARTER), h)[0], axis=1)
        consensus["target_q"] = consensus.apply(
            lambda r: qadd(int(r.YEAR), int(r.QUARTER), h)[1], axis=1)
        merged = consensus.merge(realized_df, on=["target_y", "target_q"]).dropna()
        rhos.append(spearmanr(merged[col], merged["realized"]).statistic
                    if len(merged) >= 20 else np.nan)
    return rhos

EDA 3: individual forecaster skill and its distribution

The consensus masks a wide distribution of individual skill. How different are the best and worst forecasters in the panel?

The two extremes of the dataset are illustrated clearly:

UNEMP at h=0 (nowcast): almost every individual forecaster has a Spearman correlation above 0.90 with the realised rate. The median is 0.98. There is essentially no skill heterogeneity: anyone who submits a reasonable nowcast for unemployment gets the ranking right almost perfectly. Averaging these forecasters is trivially close to optimal.
CPI at h=2: the distribution of individual correlations is centred close to zero, with a wide spread. Some forecasters have positive correlation, some have negative correlation. The median is around 0.09. This is the canonical “hard to forecast” regime: even the best-performing panelists are barely better than noise, and combining noise does not produce signal.

The intermediate cases (UNEMP h=2 and CPI h=0) are where the combination question is most interesting.

def individual_rhos(var, horizon, panel_df, realized_df, min_quarters=15):
    """Distribution of per-forecaster Spearman ρ with the realised outcome."""
    col = f"{var}{horizon + 2}"
    rows = []
    for _, row in panel_df.iterrows():
        sy, sq = int(row["YEAR"]), int(row["QUARTER"])
        ty, tq = qadd(sy, sq, horizon)
        if pd.notna(row[col]):
            rows.append({"target_y": ty, "target_q": tq,
                         "ID": int(row["ID"]), "forecast": row[col]})
    df_fc = pd.DataFrame(rows).merge(realized_df, on=["target_y", "target_q"])
    rhos = []
    for iid, grp in df_fc.groupby("ID"):
        if len(grp) >= min_quarters:
            rhos.append(spearmanr(grp["forecast"], grp["realized"]).statistic)
    return rhos

EDA 4: is individual skill persistent?

If some forecasters are systematically better than others, a combination method that upweights them should outperform equal-weighting. But is there persistent skill?

For unemployment, skill is strikingly persistent: a forecaster who performs well at h=1 also performs well at h=2. The Spearman correlation between a forecaster’s skill rank at h=1 and their skill rank at h=2 is very high. This makes intuitive sense: forecasters who understand the persistence structure of unemployment do well at all horizons.

For real GDP growth, the picture looks superficially similar (skill persistence ρ = 0.84, nearly as high as unemployment’s 0.87), but the distribution of skill is fundamentally different: many RGDP forecasters have near-zero or negative Spearman correlations with the realised outcome at both horizons, so the panel is centred near zero rather than near one. What the persistence coefficient tells us is that whatever skill or bias a forecaster has, it is durable across horizons; it does not tell us that skill levels are high. This aligns with the broader finding in the forecasting literature that RGDP growth is close to a martingale over quarterly intervals: most forecasters have little genuine skill, and those who do are consistently skilled, while those with systematic biases remain biased.

The existence of persistent skill for unemployment suggests that combination methods which exploit skill differentials could add value, at least in principle. Whether they do in practice depends on how reliably those differentials can be estimated from finite data.

EDA 5: the cross-horizon redundancy structure

When we combine forecasts at different lead times (e.g., the nowcast alongside 1-, 2-, 3-, and 4-quarter-ahead consensus forecasts), the signals are not independent. How correlated are they?

The correlation between consecutive horizons is very high: the nowcast and 1-quarter-ahead consensus correlate at 0.87; the 3Q and 4Q-ahead forecasts correlate at 0.91. Even the least-correlated pair, nowcast and 4Q-ahead, has a Spearman correlation of 0.75. This redundancy structure has a direct implication for combination: a simple average of the five lead-time forecasts dilutes the most informative one (the nowcast) with four highly redundant, less accurate versions. An optimal method should recognise this and concentrate weight on the most informative signal.

The redundancy is not accidental. Forecasters form their views about the current quarter first, then extrapolate to future quarters using a model that is largely anchored to the current forecast. A forecaster’s 4Q-ahead view of GDP growth is typically little more than their nowcast adjusted for mean reversion.

# Compute cross-horizon Spearman correlation matrix (pooled across all 8 variables)
# Using the prebuilt panel wide format: one row per (var, target_y, target_q)
df_wide = pd.read_parquet("spf_panel_wide.parquet").reset_index()
lead_cols = ["lead0", "lead1", "lead2", "lead3", "lead4"]
corr_matrix = df_wide[lead_cols].dropna().corr(method="spearman")
print(corr_matrix.round(2))
#         lead0  lead1  lead2  lead3  lead4
# lead0    1.00   0.87   0.80   0.76   0.75
# lead1    0.87   1.00   0.90   0.85   0.83
# lead2    0.80   0.90   1.00   0.90   0.87
# lead3    0.76   0.85   0.90   1.00   0.91
# lead4    0.75   0.83   0.87   0.91   1.00

EDA 6: how predictable is each variable at h=2?

The gap between the most and least predictable variables at a two-quarter horizon is enormous. Unemployment (consensus Spearman correlation 0.91 with realised) is roughly three times more predictable than CPI (0.32) and ten times more predictable than corporate profits (0.09), which is the least forecastable variable in the panel. This heterogeneity means that any pooled analysis across variables must be interpreted carefully: pooling improves statistical efficiency but conflates very different underlying environments.

EDA 7: realised values and regime shifts

The 58-year sample covers six distinct macroeconomic regimes:

1968-1979 (Great Inflation): unemployment rises and CPI climbs to double digits.
1980-1983 (Volcker disinflation): deliberate tight monetary policy drives CPI from 13% to 3% at the cost of a deep recession and peak unemployment of 10.8%.
1984-2007 (Great Moderation): low volatility in both inflation and output growth, long expansions.
2008-2009 (Global Financial Crisis): unemployment spikes from 4.4% to 10%.
2010-2019 (post-GFC recovery): slow recovery, below-target inflation throughout.
2020-2026 (COVID and aftermath): unemployment briefly hit 14.7% in April 2020, then fell sharply; CPI surged to 9% in 2022, the highest in 40 years, before gradually retreating.

This regime diversity is important for evaluating any combination method out-of-sample: a model trained on Great Moderation data may perform very differently during the inflation surge of 2021-2022. We will trace this explicitly in Part 3.

EDA 8: forecaster disagreement over time

Do professional forecasters agree more or less during economic stress?

Disagreement (cross-sectional standard deviation of 2Q-ahead UNEMP forecasts) was low and stable for most of the survey’s history, including during the Great Inflation and the Volcker recession. It shows a small visible spike at the Global Financial Crisis and then an enormous spike at COVID, by far the largest in the 58-year sample. The COVID spike is consistent with the unprecedented uncertainty about the trajectory of the labour market: no historical analogue existed for a pandemic-induced shutdown followed by an equally sharp reopening. The absence of elevated disagreement during the 1970s-1980s recessions is itself informative: unemployment is persistent enough that even during those turbulent episodes, forecasters converged on similar near-term paths.

This pattern has a direct implication for combination: during periods of high disagreement, the cross-sectional distribution of forecasts carries more information (some forecasters may have genuinely different information sets or models), making it potentially more valuable to weight them unequally. During periods of low disagreement, equal-weight is nearly optimal by symmetry.

What the EDA tells us

Before fitting any model, the data already answers several questions:

1. The combination problem is not trivial. Individual forecasters vary enormously in skill, depending on the variable and horizon. UNEMP nowcasters are essentially all equally skilled; CPI medium-term forecasters are all essentially noise. The interesting combination problem lives in the middle: variables and horizons where some skill exists, some heterogeneity exists, and some redundancy across signals must be managed.

2. The cross-horizon structure is the main design lever. The five lead-time forecasts of the same variable are a natural set of signals: they are all targeting the same outcome, they have a structural skill hierarchy (nowcast is always best), and they are highly correlated with each other. This creates the exact redundancy that a well-designed combination method should exploit.

3. Panel turnover limits named-forecaster experiments. The 58-year span is too long for a balanced panel of named forecasters. Any long-run analysis must use the consensus. For experiments on the combination puzzle (same-horizon, named-forecaster combinations), we are limited to smaller panels where a few high-coverage individual forecasters overlap for the same variable, horizon, and target quarter.

4. Regime diversity is substantial. A combination method that works during the Great Moderation may fail during an inflation surge. Part 3 checks whether the main rankings remain stable through time using rolling out-of-sample performance summaries.

What comes next

Part 2 builds the mathematical framework. We separate two approaches: modelling the joint dependence of outcomes and forecasts, and modelling the joint dependence of forecast errors. The Gaussian outcome-forecast copula reduces to OLS / best linear prediction on probit ranks; the Gaussian error model gives the more classical GLS-style precision-weighted combination. We also introduce vine copulas for both approaches and use synthetic data to show when their extra flexibility helps.

Part 3 puts the resulting method families on the SPF data: simple baselines (persistence, Lead 0, equal-weight, trimmed mean, inverse MSE), outcome-forecast methods (OLS/rank regression, Gaussian copula, vine copula), and error-dependence methods (error GLS, shrinkage variants, and error-copula diagnostics). It reports the walk-forward horserace, revision-sensitivity checks, rolling out-of-sample performance, and named-forecaster case studies.

Data: SPF Individual Forecasts, Federal Reserve Bank of Philadelphia.

Who Competes with Whom? Mapping Hong Kong's Financial Competitor Network from Labour Market Data

Wed, 01 Apr 2026 00:00:00 +0000

Who Competes with Whom? Mapping Hong Kong’s Financial Competitor Network from Labour Market Data

When Citadel loses a portfolio manager to Point72, both firms understand the competitive stakes. The SFC register records when the licence at Citadel was cancelled and when a new one at Point72 was activated. The gap between them is a legal mechanism (the non-compete clause) but also a data signal. For moves between rival hedge funds, 13% of professionals wait more than a year before their next licence activates. This post is about what that signal, aggregated across thousands of transitions over 20 years, reveals about competitive structure in Hong Kong’s financial industry.

tl;dr Using 20 years of regulatory data from Hong Kong’s Securities and Futures Commission (SFC), we construct a firm-firm competitor network for approximately 5,000 financial companies, without relying on self-reported disclosures or analyst judgment. The key insight: when employees move between firms, they leave a behavioural trace of competitive relationships. But not all moves are equal. A move from Goldman Sachs to a hedge fund is career progression; a move from Citadel to Point72 is genuine rivalry. We identify 315 hedge funds, trace 918 person-transitions between them, and map a competitor network of 2,387 firms with 14 distinct communities.

Context: The problem with competitor data

Commercially available competitor datasets are either self-reported (a company says who it thinks competes with it) or constructed by analysts based on SIC codes and business descriptions. Both approaches have well-known limitations: self-reported data reflects strategic disclosure choices rather than ground truth, SIC codes are coarse proxies for actual business overlap, and private firms (which make up the vast majority of Hong Kong’s financial industry) are largely invisible to both methods.

This project started from a simple observation: Hong Kong’s Securities and Futures Commission publishes a complete, public register of every licensed financial professional since 2003, including name, firm, regulated activity types, and the exact dates their licence was active. If you know when a person was at Firm A and when they joined Firm B, you have a behavioural signal of competition that would be invisible otherwise. Firms compete for the same talent precisely because they do the same thing.

This builds directly on two prior papers: Mapping Hong Kong’s Financial Ecosystem (which used the same register to study the network structure of licensed professionals and firms) and Turnover Contagion in HK Financial Labour Markets.

Data: 519,860 licence records, 20 years, 5,000 firms

The SFC register contains 519,860 licence records spanning 2003–2024, covering 121,883 unique persons and 4,979 distinct firms. Each record specifies the regulated activity type (RA1 through RA10):

RA	Description
RA1	Dealing in Securities
RA2	Dealing in Futures Contracts
RA4	Advising on Securities
RA6	Advising on Corporate Finance
RA9	Asset Management
RA10	Providing Credit Ratings

Large financial groups typically appear as multiple distinct legal entities in the register: Goldman Sachs (Asia) L.L.C., Goldman Sachs Futures Co. Ltd., and Goldman Sachs Private Wealth Management Ltd. are three separate SFC registrants, but treated as a single competitive entity in our analysis. We consolidate these into 4,576 parent entities using Claude Haiku to identify ultimate parent relationships, followed by two targeted merge passes based on inspection of the highest-similarity pairs. All analysis from this point forward operates at the parent-entity level.

Methodology: Three signals, one score

We construct the competition score between every pair of parent entities as a weighted combination of three signals. Each captures a fundamentally different dimension of competition and they are not interchangeable.

\[C[i,j] = 10 \times \left( \frac{1}{3} \cdot \tilde{J}[i,j] + \frac{1}{3} \cdot \tilde{F}[i,j] + \frac{1}{3} \cdot \tilde{E}[i,j] \right)\]

Signal 1 — Activity Overlap (Jaccard): potential competition

For each parent entity, we compute a binary vector over the 10 regulated activity types. The Jaccard similarity between two firms measures the overlap of their regulatory footprints:

\[J[i,j] = \frac{|A_i \cap A_j|}{|A_i \cup A_j|}\]

Two firms that both hold RA1, RA4, and RA9 score J=1.0. This is a structural signal: do these firms operate on the same regulatory fields? It has full coverage (computable for all 8.3M pairs) and is stable over time. Its main limitation is coarseness: a Chinese retail broker and a global investment bank both hold RA1, but they serve entirely different clients.

Signal 2 — Normalised Employee Flow: revealed competition

For each person, we detect sequential transitions: a tenure at Firm A ending within a window of a tenure at Firm B starting. We aggregate and normalise by firm size:

\[F[i,j] = \frac{\text{flow}_{i \to j} + \text{flow}_{j \to i}}{\sqrt{n_i \cdot n_j}}\]

This is a behavioural signal: do firms actually compete for the same people? It is observable from the public register regardless of whether either firm chooses to disclose competitive relationships.

The key methodological choice is the transition window. We use 730 days (2 years) as the cap, which ensures that moves constrained by 12–24 month non-competes are not lost. A 180-day window (the standard in labour mobility studies) would miss nearly 40% of IB→HF transitions and 28% of HF→HF transitions that occur in the 6–24 month gap range. Using 730 days adds 39% more observed transitions overall compared to 180 days.

Despite its informational value, flow covers only approximately 1% of all pairs. The vast majority of the network topology derives from Jaccard and embedding. Flow is most valuable for identifying and ranking the most competitive specific relationships, not for defining the network boundary.

Signal 3 — Embedding Cosine Similarity: semantic competition

We generate a short description of each firm using Claude, embed all 4,576 descriptions with all-mpnet-base-v2 (768-dimensional SBERT), and compute pairwise cosine similarity. This semantic signal adds value for firms whose regulatory footprints differ but whose business is clearly similar: two quant funds that may hold different RA combinations but describe themselves in near-identical terms.

What each signal can and cannot see

	Jaccard (structural)	Flow (revealed)	Embedding (semantic)
Coverage	All pairs	~1% of pairs	All pairs
What it captures	Regulatory overlap	Actual talent competition	Descriptive similarity
Blind spots	Ignores strategy, client base, geography	Sparse; finite window misses very long career breaks	Depends on LLM description quality
Best use	Broad network topology, community detection	Ranking competitive intensity; validating specific relationships	Niche discrimination within broad activity types

A regression on 4,000 LLM-scored firm pairs confirms that the signals are complementary rather than redundant. SHAP feature importance:

Feature	SHAP contribution
Jaccard similarity	0.760
Same firm type	0.365
Embedding similarity	0.344
Flow (normalised)	0.014

The low SHAP for flow reflects that it is independent of what the LLM uses to judge competition: it measures revealed behaviour that is invisible to any structural analysis. Equal weights are an arbitrary design choice. Jaccard dominates the LLM-based regression, but flow and embedding each provide orthogonal signal that would be lost under SHAP-proportional weighting.

A few concrete examples of signal alignment and divergence:

Pair	Score	Raw flow	Norm flow	Reading
Goldman Sachs / Morgan Stanley	5.74	151	0.020	Both signals align: canonical IB peers
UBS / Credit Suisse	5.42	179	0.045	Both signals align: full-service bank peers
Penjing Asset Management / Sigma Capital	6.16	18	0.455	Highest normalised flow in the dataset: two small AM firms with intense bilateral exchange relative to size
CMB International / CMBC International	6.39	0	0.000	Two distinct Chinese bank-affiliated IB arms with near-identical activity profiles; genuine competitors, zero observed talent crossing

Network structure: 14 communities, modularity 0.622

Applying a minimum firm size of 10 persons and competition score threshold τ=4, the network has 2,387 nodes and 317,850 edges. Louvain community detection (modularity=0.622) finds 14 communities:

Community	Size	Representative members	Interpretation
0	706	Goldman Sachs, Morgan Stanley, UBS, BofI Securities	Full-service investment banks and their execution arms
1	412	HSBC, Bank of China, ICBC, Orient Securities	Bank-affiliated commercial banks and their securities arms
2	391	Phillip Securities, KGI, Guangdeng Futures	Retail broker-dealers and futures intermediaries
3	256	Point72, Citadel, Balyasny, Millennium, BFAM	Hedge funds
4	208	Zhongtai International, ICBC International Securities, CCB International	China-affiliated mid-tier investment banks
5	122	Phillip Capital, Haitong International Futures, BofI Securities (formerly Merrill Lynch)	Futures-heavy broker-dealers
6	102	CPPIB, Computershare, smaller retail brokers	Smaller retail and custody-focused firms
7	60	Ping An, PricewaterhouseCoopers, Lego Corporate Finance	Insurance, audit, and corporate advisory
8	53	Zhongtai International Capital, RHB Capital, Fortune Financial Capital	Mid-tier regional banks and boutique IBs
9	34	FTL Wealth Management, Thornton Global, Wisdom Wealth	Independent wealth managers and IFAs
10	28	CES Forex, Emperor International Exchange, Rakuten Securities	Forex and retail FX platforms
11	8	S&P Global Ratings, Moody’s, Fitch	Credit rating agencies
12	5	Liquidnet, OSL Digital Securities, Hash Blockchain	Crypto and digital assets
13	2	Celestial Finance, Lei Shing Hong Finance	Residual micro-entities

The modularity of 0.622 is high: communities are tightly internally connected and sparsely interconnected. Three global network properties are worth noting:

Density = 0.127: 12.7% of all possible firm pairs exceed the competition threshold τ=4, relatively dense for an industry competitor network.
Clustering coefficient = 0.825: if Firm A competes with B, and B competes with C, A almost certainly competes with C. Competition in financial services is highly transitive, as activity overlap propagates.
Betweenness centrality: dominated by broker-dealers and mid-tier Chinese-affiliated firms, not the global investment banks. This is counterintuitive but structurally sensible: global IBs are so densely interconnected within their own community that they rarely serve as bridges. Local broker-dealers, with licences spanning retail, futures, and asset management, sit at the boundary between communities and act as connective tissue across the network.

Historical fingerprints: M&A, collapses, and market exits

The SFC register records every major institutional event that reshuffled Hong Kong’s financial workforce over two decades. Mergers, acquisitions, collapses, and strategic retreats all leave a characteristic signature: a concentrated burst of same-direction transitions within a narrow window.

Year	From	To	Persons	Event
2008	Lehman Brothers	Nomura	423	Nomura acquisition of Lehman’s Asia-Pacific business
2010	Merrill Lynch	BofI Securities	217	Bank of America / Merrill Lynch integration and HK rebrand
2017	Tung Shing Securities + Futures	SinoPac Securities	155	SinoPac acquisition of Tung Shing’s HK brokerage
2005	Prudential Assurance	BOC International	148	Prudential restructured its HK securities operations
2023	Credit Suisse	UBS	45	UBS emergency rescue acquisition (March 2023)
2012	Royal Bank of Scotland	China Galaxy Securities	39	RBS exit from Asia, sold to Chinese state-owned buyer
2012	Crédit Agricole	China Galaxy Securities	26	Crédit Agricole’s HK business absorbed by China Galaxy
2008	Lehman Brothers	Neuberger Berman	19	Neuberger Berman (Lehman’s AM arm) carved out independently

The 2008 Lehman event is the most dramatic in the dataset: 423 of 470 total Lehman to Nomura transitions occurred within the calendar year of the collapse. The 2012 entries reflect a structural shift, as RBS and Crédit Agricole both withdrew from Hong Kong and were absorbed by the same Chinese state-owned buyer, reflecting the broader retreat of Western banks from Asia operations in that period.

The Hedge Fund Ecosystem

Scale: 7,616 professionals, 6.2% of the register

Across the full 20-year register, 7,616 persons (6.2% of 121,883 total) held at least one licence at a hedge fund. The HF workforce has grown 14-fold since 2003 (182 to 2,543 active professionals by end of 2023):

Growth was rapid through the 2000s (quadrupling 2003–2008), resilient through the GFC, and plateaued in 2022–2023. The plateau reflects several converging pressures: the prolonged China equity bear market from late 2021 onwards, regulatory actions affecting offshore China-focused funds, the unwind of leveraged macro China positions, and rising interest rates redirecting global LP capital away from Asia-focused alternatives. Note: 2024 is excluded as a partial year.

Identifying 315 hedge funds

The SFC has no “hedge fund” licence category. A fund running a long/short equity book and a Vanguard ETF product both hold RA9 (Asset Management). We classify all 1,113 firms in the asset manager / other / PE / wealth management categories using Claude Sonnet, with only firm name, Chinese name, and regulated activity types as input.

The result: 315 hedge funds identified. A second pass classifies each by primary strategy:

Long/short equity dominates by count (183 funds), reflecting Hong Kong’s role as the primary hub for Asia and China-focused equity strategies. Multi-strategy platforms are fewer (14 funds) but disproportionately large: Millennium (372), Point72 (295), and Citadel (259) are the three largest by registered professionals.

Survival: 244 active, 69 closed, median lifespan 10.5 years

Of the 315 HFs, 244 remain active and 69 have closed. The median lifespan of closed funds is 10.5 years: these are not failed startups but mature funds that ran their course. Closure rates are highest for 2006–2007 cohorts (44–60%), likely GFC casualties.

Among recent closures: Adamas Asset Management (2023, 10 years), Snow Lake Capital (2023, 11 years), Myriad Asset Management (2022, 11 years), and Azentus Capital Management (9 years).

Goldman Sachs as training ground

Investment banks export talent to hedge funds at a 3.32× ratio relative to the reverse (1,271 IB to HF vs 383 HF to IB). This is career progression, not competition. Goldman Sachs sends 185 persons to hedge funds over the sample, the single largest feeder in the dataset, followed by Credit Suisse (158, combining all group entities), Morgan Stanley (127), Citigroup (77), JPMorgan (66), and UBS (64).

Investment banks lead with 30.4% of all HF inflows, followed by HF-to-HF moves (22.0%, covered in the next section), asset managers (21.1%), and broker-dealers (17.8%). The figure above shows only non-HF feeders; HF-to-HF transitions are analysed separately below.

The broker-dealer channel (17.8%, 745 persons) deserves attention. Broker-dealer professionals (equity sales traders, research analysts, derivatives structurers) move to hedge funds as execution talent: people who know how to work markets, manage relationships with prime brokerage desks, and execute strategies efficiently. These transitions tend to be faster, with a median non-compete gap of 79 days, shorter than IB to HF moves at 96 days.

HF-to-HF competition: the inner circle

918 persons transitioned between HF pairs over the 20-year sample (730-day window). The top undirected pairs:

Firm A	Firm B	Persons
Adamas Asset Management	Gen2 Partners	15
Millennium Capital Management	Segantii Capital Management	13
Gentleman Capital	Goldhorse Capital	13
Millennium Capital Management	Polymer Capital	12
Millennium Capital Management	Point72	12
Balyasny	Point72	11
Dymon Asia	Millennium Capital Management	11
Balyasny	Millennium Capital Management	10

The most active competitive cluster is the Balyasny–Millennium–Point72 triangle, the highest-weight triplet in the dataset. Millennium appears in more top bilateral flow pairs than any other fund, reflecting its model as a multi-PM platform with high hiring and attrition across the ecosystem.

The network reveals two distinct competitive topologies. The global multi-strategy platforms (Millennium, Balyasny, Point72, ExodusPoint, Schonfeld) form a dense triangle core: they compete for the same PM talent pool, and the competition is mutual. The HK-homegrown funds (Gen2, Adamas, Gentleman, Goldhorse) show intense bilateral competition without triangular clustering, concentrated within strategy niches. Segantii is a hub-without-triangles: bilateral flows with 21 different funds spanning global multi-strats, macro, credit, and event-driven, consistent with an event-driven strategy whose talent pool overlaps many adjacent strategies.

Talent magnets and exporters

Net flow (inbound minus outbound HF→HF transitions) reveals a prestige hierarchy:

The strongest net importers are Millennium (+30), Polymer Capital (+18), Adamas (+15), Gentleman Capital (+13), and Dymon Asia (+13). Millennium’s dominance is striking: with 91 inbound and 61 outbound HF to HF moves, it is the platform of choice for experienced hedge fund professionals across strategies.

The largest net exporters are Sculptor Capital (−15), Goldhorse (−13), LIM Advisors (−13), Azentus (−11), and Point72 (−10). Several closed funds appear here, consistent with net outflow as a leading indicator of institutional stress. Point72 as a net exporter is notable: its single largest outflow destination is Millennium (10 persons), followed by Balyasny (6) and Schonfeld (4). PMs whose capital is reallocated typically move to another multi-strategy platform rather than exiting the industry. Point72 functions as a training and filtering ground within the multi-strat ecosystem.

Tenure: the whole industry is high-turnover

A common assumption is that hedge fund tenures are uniquely short and volatile. The data does not support this:

Firm type	Median tenure	<1 year	<2 years
Hedge Fund	1.9 years	29%	52%
Investment Bank	2.0 years	28%	51%
Asset Manager	2.0 years	29%	50%
Broker-Dealer	1.8 years	33%	54%

Median tenures are nearly identical across all firm types. The entire Hong Kong financial industry is a high-turnover labour market.

Two caveats on interpretation. First, SFC records are at the legal entity level: a professional who spends eight years at Goldman Sachs may appear as two or three separate tenures across different Goldman Sachs entities, compressing apparent tenure. Second, SFC-registered roles are disproportionately front-office and senior (portfolio managers, traders, analysts, relationship managers), roles with structurally higher turnover than back-office functions.

The gap between roles: where non-competes show up

When a professional leaves one firm and joins another, the gap between their last licence deactivation and next licence activation reflects the nature of the move.

Transition	Median gap	>180 days	>1 year
IB to IB	62 days	22.3%	13.8%
BD to HF	79 days	27.1%	16.2%
IB to HF	96 days	33.5%	19.7%
HF to HF	100 days	31.7%	13.1%

(Positive gaps only, transitions capped at 3 years.)

The medians are broadly similar across transition types, ranging from 62 to 100 days. The fraction waiting more than a year varies more: IB to HF stands at 19.7%, compared to 13.1% for HF to HF. Whether this reflects more aggressive IB non-compete enforcement on buy-side departures, or other factors such as more senior roles being involved in IB to HF moves, is difficult to disentangle from this data alone.

Breaking down IB to HF transitions by SFC experience at departure does not show a simple seniority gradient. The >1 year fraction is 24.5% for the least experienced tier and 21.7% for the most experienced, with mid-career professionals (3–6 years SFC experience) showing the shortest gaps at 18.3%. This is consistent with the SFC experience measure being a lower bound: international hires with prior careers outside Hong Kong appear junior by this metric even when they are in fact senior. Given this ambiguity, the seniority breakdown should be read with caution.

HF to HF moves (100 days median) show a lower long-gap fraction (13.1%) than IB to HF. This partly reflects that a share of HF to HF transitions occur when a fund closes, circumstances where the former employer has no competitive interests to enforce.

Career trajectories: how much experience before the HF?

For the 7,616 persons who ever joined a hedge fund, 48.8% have no prior SFC record. This reflects several situations: fresh graduates recruited directly by funds, foreign hires whose prior careers were in other jurisdictions, and professionals coming from roles in Hong Kong that do not require SFC registration (quantitative researchers, for instance, are often not registered unless they hold a portfolio management role). Among those who came from other registered firms, median prior SFC experience ranges from 4.7 years (private equity) to 6.2 years (insurance), with IB professionals joining after approximately 4.9 years. All figures are lower bounds; the pre-2008 cohort is systematically undercounted.

Validation

Four consistency tests confirm the network captures meaningful competition:

1. Skill-specificity: employees moving between competitor pairs carry significantly more similar regulated activity licences (actType Jaccard 0.62 vs 0.50; Mann-Whitney p≈0).

2. Flow intensity: competitor pairs have 1.4× higher normalised flow than non-competitor pairs (p≈10⁻²⁴). The signal is real but modest, with only approximately 1% of pairs having any observed flow.

3. Firm-type consistency: 52.9% of competitor pairs are the same firm type, vs. a 26.9% random baseline (χ²=109,254, p≈0).

4. Tenure signature: competitor-pair movers have longer pre-move tenure (701 vs 631 days median; p≈0), reflecting that movers to direct competitors tend to be more senior, and senior departures attract more scrutiny and longer notice periods.

Limitations

Career start lower bound: The register starts in 2003. Experience estimates are significantly downward-biased for professionals already in the industry by then.
730-day window: Chosen to capture non-compete-constrained flows. True multi-year career breaks are rare at the senior level and should not be counted anyway, so this window is a pragmatic rather than conservative choice.
HF classification at the small end: Some of the 315 HFs, particularly those with generic names and few registered professionals, may not be genuine hedge funds.
Embedded HF units: Strategies such as equity stat-arb units within large AM groups cannot be separated at the parent-entity level.
Absence of major quant funds: Several globally prominent quantitative funds (e.g. Squarepoint Capital) have no SFC-licensed entity in Hong Kong, possibly operating through Singapore. The quant segment is likely the most incomplete.
Team hires and spinoffs: Both Qube Research & Technologies (QRT) and Y-Intercept originated from Credit Suisse’s SMG group. QRT had 12 direct CS to QRT moves in 2018 alone. Y-Intercept shows only 4 direct CS to YI moves in the register; its founding team may have come from Credit Suisse offices outside Hong Kong, which would leave no HK SFC record.
Selective registration: Quantitative researchers may or may not be SFC-registered. Some firms reportedly de-register research staff precisely because licence records are public and signal headcount changes to competitors.
Static analysis: All results use the full 2003–2024 window. Year-by-year tracking of how competitive relationships form and dissolve is the natural extension.

Conclusion

Twenty years of regulatory data from Hong Kong’s SFC reveals a rich competitor network that no commercial data provider currently constructs. Labour mobility is a behavioural signal of competition, but the signal must be read carefully. Not every move is a defection.

Key findings:

The HK financial competitor network has 14 communities (modularity=0.622) with clear alignment to firm type; betweenness centrality is dominated by broker-dealers, not global IBs
Investment banks are training grounds, not competitors: Goldman Sachs sends 185 persons to hedge funds over 20 years, the most of any single institution, followed by Credit Suisse (158) and Morgan Stanley (127)
315 hedge funds identified (244 active, 69 closed); Millennium appears in more top bilateral flow pairs than any other fund, defining the competitive core of the multi-strategy ecosystem
IB to HF transitions show the highest fraction of moves exceeding one year (19.7%), with plausible explanations including non-compete enforcement on senior buy-side departures and the seniority profile of these moves; HF to HF stands at 13.1%
48.8% of HF entrants have no prior SFC record, reflecting direct recruitment from graduate programmes, overseas hires, and professionals coming from roles not requiring SFC registration

The methodology is fully reproducible from the SFC public register. A prepared extract of the register is available as a public dataset on Kaggle. The academic paper is in preparation.

This work builds on Mapping Hong Kong’s Financial Ecosystem and Turnover Contagion in HK Financial Labour Markets.

This is an academic research project conducted in collaboration with Abdulla Al Ketbi, in the context of his PhD research on complex networks, explainable AI, and complexity science. The results presented here are preliminary and will be incorporated into his PhD manuscript and associated publications. This post does not constitute investment advice and has no connection to any employer or commercial entity.

EMNLP 2025 in Suzhou

Sun, 30 Nov 2025 00:00:00 +0000

EMNLP 2025 in Suzhou

Disclaimer: The research discussed in this post is entirely unrelated to my professional responsibilities. The work described here reflects independent academic activity that I pursue out of personal interest in modern machine learning research. All papers referenced below have been officially approved for publication and contain no proprietary or confidential information from my current employer. The material presented should not be interpreted as investment advice, financial analysis, or as having any applicability to my employer’s activities. Its relevance is strictly limited to the advancement of empirical methods and scientific understanding in machine learning.

Motivation

This year at EMNLP 2025 in Suzhou, my colleague Khaled Al Nuaimi and I attended the conference so that Khaled could present his paper on Evasive Answers in Financial Q&A, and also to explore current R&D trends in empirical NLP.

While walking through the poster sessions, we saw a dozen of papers closely related with our recent contributions and joint research program with Khalifa University, where I co-supervise the PhD work of my colleagues:

Across these doctoral projects, we are building a coherent, multi-layer research program at the intersection of:

Complex networks & complexity science (financial labour networks, institutional networks, contagion processes)
Feature importance & explainable AI (alignment between model predictions and explanations in LLMs & Graphs (Networks) ML models)
Multimodality: audio + text (tone, prosody, evasiveness, spoken Q&A behaviour)
Financial Q&A and behavioural analytics (linguistic deception, psychological discourse strategies, managerial behaviour)
Agent-based modelling (ABM) with explainable agents (replacing zero-intelligence coin-flipping agents with more realistic, behaviourally grounded AI agents)

At EMNLP 2025, we encountered many papers that map surprisingly well onto this agenda, in some cases reinforcing our assumptions, in others providing tools or methodologies we can directly adopt.

Below is a curated summary of the key papers we saw and how they connect to our ongoing work.

Relevant (to us) EMNLP 2025 Papers

Zero-Shot Extraction of Stock Relationship Graphs with LLMs (Zhou et al., FinNLP 2025)

paper

What the paper does:

Uses LLMs as zero-shot knowledge bases to extract rich, typed company–company relations (supplier, competitor, JV partner, investor…).
Builds multi-relational stock-market graphs from LLM outputs.
Trains GNNs (RGCN, RGAT) for stock return ranking, outperforming GICS/Wikidata graphs.

Connections to our research:

This work aligns most closely with Abdulla’s line of research. Beyond reconstructing networks of past co-employment directly from SFC data, we could extend the financial ecosystem by using LLMs to infer missing inter-institution relationships. For example: competition, parent–subsidiary structures, shared ownership, or other strategic ties.

Adding these extra edges would enrich Abdulla’s labour-contagion models by capturing sub-industry exposure, competitive pressures, organizational linkages, and cross-firm risk propagation. In short, it gives us a more complete picture of how information, shocks, or turnover might spread across the ecosystem.

From the perspective of our agent-based modelling (ABM) vision, a multi-relational LLM-inferred graph provides a richer structural layer for agents to operate on. Employee-agents could factor in market competition or firm relations when deciding whether to move to a new opportunity, while firm-agents could incorporate similar relational cues when adjusting headcount or assessing talent flows. This creates a more behaviourally realistic and environmentally aware agent society than classic zero-intelligence ABMs.

Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey (Chih-Kai Yang et al.)

paper

What the paper does:

A taxonomy for evaluating audio-language models (LALMs), organized along four axes:

Auditory awareness & processing
Knowledge & reasoning
Dialogue-oriented ability
Fairness, safety & trustworthiness

It also highlights several gaps in current audio-NLP practice:

No clean separation between content and paralinguistic cues
Weak robustness evaluation (noise, speaker variation, accents)
Limited cultural/linguistic diversity in benchmarks
Limited evaluation of dialogue quality and response appropriateness

Connections to our research:

Hamdan’s first research contribution: Residual Speech Embeddings

A natural question is: What differentiates LLMs from LALMs? In principle, LALMs should integrate auditory information in addition to text: capturing tone, speaker traits, emotional signals, and other paralinguistic cues that are lost in transcripts.

However, the survey highlights a consistent limitation: current LALMs exhibit very limited auditory awareness. They often fail to detect speaker changes, emotional shifts, or mismatches between lexical content and paralinguistic delivery. In practice, their behaviour is much closer to text-only LLMs than to genuine audio-language models.

These observations align closely with what we have seen empirically. This is precisely the motivation for Hamdan’s work on Residual Speech Embeddings, where we demonstrate that self-supervised speech models tend to over-emphasize lexical information at the expense of paralinguistic cues.

Our approach explicitly removes the lexical component from the speech embedding space, leaving a representation that captures tone and other non-textual characteristics. Empirically, we show that this improves performance on tone-classification tasks, confirming that isolating the paralinguistic signal is beneficial.

For reference:

This EMNLP paper essentially validates the direction of our work: the field currently lacks benchmarks that disentangle content from paralinguistic cues, and residual speech embeddings provide a practical step toward filling that gap.

It is also worth noting that Hamdan’s work naturally connects to Khaled’s research on financial Q&A. One of Khaled’s recent contributions focuses on detecting evasive answers in earnings calls and FOMC press conferences, based purely on the textual transcripts. A logical next step is to extend this analysis to the audio recordings themselves.

Working directly from audio opens new questions that transcripts cannot capture:

Are there identifiable paralinguistic cues associated with evasive answers?
Do these cues vary depending on the type of evasive tactic employed?

Hamdan’s residual embeddings provide exactly the kind of representation needed to explore these audio dimensions of evasiveness in a controlled and interpretable way.

From a broader perspective, this line of work also hints at a longer-term, more speculative direction for our agent-based modelling (ABM) research. If we can reliably disentangle and quantify paralinguistic cues, it becomes conceivable—though certainly not immediate—to develop speech-enabled agents whose behaviour is influenced by vocal signals such as stress, confidence, or hesitation. While this is not a short-term objective, it illustrates how multimodal modelling could eventually lead to more realistic simulations of financial decision-making dynamics.

The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (Fan et al.)

paper

What the paper does:

Embedding-based similarity metrics between text sequences can be influenced not just by the content dimensions we most care about, but can also be biased by spurious attributes like the text’s source or language. These document confounders cause problems for many applications, but especially those that need to pool texts from different corpora.

To mitigate this, the authors introduce Linear Concept Erasure (LCE), a linear projection method that identifies the subspace corresponding to these confounders and removes it, while preserving the semantic core of the representation. Their main findings are:

cross-corpus clustering and retrieval improve after removing the confounder subspace;
downstream semantic tasks remain essentially unchanged, which is the expected outcome (semantics are preserved, and only nuisance variation is removed);
the method is simple and computationally light.

In practice, this is a form of representation surgery: identify the part of the embedding space capturing the unwanted factor, project it out, and keep the rest.

Connections to our research:

This paper reinforces an intuition that also underpins Hamdan’s work on Residual Speech Embeddings: To clean the representation space by removing unwanted components.

There is a clear structural similarity between the two approaches:

Residual speech embeddings remove the linear component of speech that can be predicted from text, isolating paralinguistic information;
LCE removes the linear component of text embeddings that corresponds to medium/domain/style, isolating semantic information.

In both cases, the idea is the same:

Identify a linear subspace corresponding to an unwanted factor, remove it, and work with the cleaned embedding.

We used the term residual embedding simply by analogy with “residual returns” in quantitative finance, where regression residuals are often used to remove unwanted exposures. LCE formalizes a closely related idea on the text side, but using discriminative subspace estimation rather than cross-modal regression.

This opens several possibilities relevant to our research program:

removing speaker identity or channel artifacts from audio embeddings;
removing demographic confounders in textual or multimodal representations (cf. our name-based demographics study);
isolating emotional or deceptive cues in financial Q&A;
making embeddings cross-corpus comparable by suppressing stylistic and format-specific variation across sources like broker reports, earnings calls, and regulatory filings.

Overall, LCE provides an elegant confirmation that linear subspace removal, the core idea behind Hamdan’s residual embeddings, is a powerful and general approach for disentangling competing factors in representation spaces.

Finally, This echoes an intuition I had years ago when doing topic modelling on business descriptions: the dominant axis of variation was almost always the industry sector, which is obvious but not very informative. Removing that first dimension (for example by residualizing to sector embeddings) may be necessary to reveal more interesting structure.

CafGa: Customizing Feature Attributions to Explain Language Models (Boyle et al.)

paper

What the paper does:

CaFGa proposes an attribution framework that lets users control the granularity of explanations for LLM decisions. Instead of relying on token-level heatmaps, typically noisy, unstable, and expensive, CaFGa groups text into interpretable segments (sentences, clauses, rhetorical units) and computes attributions at this coarser, more meaningful level.

A key component is the use of perturbation–fidelity curves to assess whether the explanation is faithful to the model’s actual decision process. The idea is simple: remove or mask the high-attribution segments and measure how the prediction changes. Faithful explanations yield steep fidelity curves; unfaithful ones remain flat.

The authors show that coarse-grained explanations tend to be more stable, more readable, and more faithful to model behavior than traditional word-level attributions.

Connections to our research:

Although Saeed’s credit-risk classifier operates on tabular features rather than text, CaFGa suggests a direct methodological analogue: Group features into semantic blocks (employment, income, liabilities, credit history) and compute block-level attributions. Perturbation–fidelity curves would then allow us to test whether the explanation aligns with the model’s true decision boundary. This complements our ongoing work on explanation alignment, where we repeatedly observe that LLM-generated natural-language explanations diverge from the model’s actual reasoning. CaFGa provides a principled way to quantify this divergence.

The paper is also highly relevant to Khaled’s work on evasive answers in financial Q&A. CEO (and CFO/COO) responses in earnings calls naturally decompose into meaningful discourse segments: direct answers, hedges, topic shifts, credibility boosters, vague qualifiers, and so on. CaFGa’s segment-level attribution framework fits this structure almost perfectly. It would allow us to quantify which parts of an answer contribute most to the model’s evasiveness prediction, and whether those attributions are faithful. This could naturally lead to a follow-up study on segment-level explanations for evasive answer detection.

Beyond academic value, such a tool would be practically useful for financial analysts during live Q&A sessions or one-to-one calls with management. Highlighting the segments most responsible for an “evasive” classification, together with an indication of in which direction the answer is evasive (topic shift, hedging, excessive reassurance, lack of specificity), could help analysts decide when to push further, and on which aspect of the response to follow up.

LLMs Don’t Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations (Mayne et al.)

paper

What the paper does:

This paper investigates whether LLMs understand their own decision boundaries when used as classifiers. The authors evaluate a simple but revealing setup: small tabular datasets (2–4 features) are converted into natural-language profiles, and the LLM is asked to (i) make a prediction and then (ii) provide a self-generated counterfactual explanation (SCE): “What is the smallest change to this profile that would flip your decision?”

Crucially, authors can enumerate the full input space, so they know exactly where the decision boundary lies.

They evaluate two properties:

Validity: Does the counterfactual actually flip the model’s prediction?
Minimality: Is the suggested change close to the true decision boundary?

The core findings:

When asked for any counterfactual, LLMs often produce valid but highly non-minimal edits (overshooting the boundary).
When asked for the smallest change, LLMs produce minimal-looking edits that frequently fail to flip the prediction.
No current model (including GPT-4.1, o3, Claude Sonnet 3.7) achieves both validity and minimality at the same time.

In practice: LLMs generate counterfactuals that sound correct but fail to match the true decision boundary.

Connections to our research:

This paper directly reinforces the findings from Saeed’s paper on credit-risk explanation alignment, where we show that LLM-generated explanations for credit decisions often diverge from the factors that actually drive the model’s classification. That work focused on feature-importance misalignment; this paper adds a complementary phenomenon: boundary-awareness misalignment.

Together these papers paint a consistent picture:

LLMs struggle to articulate why they made a decision (feature attributions).
LLMs also struggle to articulate how to change the decision (counterfactuals).

The methodology in this poster offers a clean way to test this in our credit-risk setting:

Use the same borrower profiles and prompt templates from Saeed’s work.
Ask the LLM to predict default vs. non-default.
Request a self-generated counterfactual: “What is the minimal change that would flip your decision?”
Parse the edited profile back to tabular form.
Re-query the model and measure validity + minimality.

Can we replicate this research in our setting?

While a direct replication of their protocol in the credit domain is rather straightforward, the more interesting direction for us is constructive: Can we help LLMs generate counterfactuals that are both valid and minimal?

This suggests a methodological extension: Adapting classical ML counterfactual engines (e.g., DiCE) on LightGBM) to identify truly minimal edits… This could lead to a more ambitious follow-up paper: “Teaching (Credit) LLMs Their Own Decision Boundaries” where the goal is not only to expose the misalignment but to improve boundary-awareness using hybrid ML–LLM pipelines (tentatively).

Conclusion

This quick wrap-up covers only a handful of the papers we came across, but they were among the most directly connected to the research lines pursued by the PhD students I’m co-supervising. Each of them sharpens or extends the work we’ve already been doing, whether on representations, audio–text multimodality, explainability, or financial Q&A behaviour.

I’m writing this just a few hours before boarding my flight to NeurIPS (San Diego). With a bit of luck, I’ll come back with another batch of papers that are equally interesting and aligned with our research program.

Serious, Sassy, or Sad? Teaching Machines to Read the Room (From Speech Embeddings)

Sun, 16 Feb 2025 00:00:00 +0000

Serious, Sassy, or Sad? Teaching Machines to Read the Room (From Speech Embeddings)

📌 A politician says ‘We are fully in control of the situation’ — but their voice is trembling. Would you trust the words or the tone? Can AI detect when speech and vocal tone don’t match?

Introduction
The experiment
Research question
tl;dr
Loading the Embeddings
Understanding the Embeddings
Implications
Validating the Findings with Supervised Classification
Short vs. Long Sentences
🔍 Key Takeaways
Discrepancy Index – A Future Research Direction

N.B. This blog post serves as a pedagogical introduction to the core concepts and methodologies explored in Hamdan Al Ahbabi’s PhD research as part of his doctoral studies at Khalifa University, under my co-supervision. It is entirely non-work-related and should not be interpreted as such. Instead, its purpose is to support and complement future presentations at academic conferences.

Introduction

In our previous blog post, Disentangling Speech Embeddings, we introduced a simple yet effective method for removing linguistic content (what is being said) from speech embeddings. Concretely, we trained a linear model to predict raw speech embeddings from the corresponding text embeddings and used the residuals (differences) as a proxy for non-linguistic vocal features. We demonstrated that the resulting “residual” embeddings retained enough vocal cue information to cluster sentences by speaker identity rather than semantic content — suggesting that these embeddings preserve vocal tone and style independently of the words spoken.

Now, we take this idea one step further. This time, we explore whether these residual embeddings can help classify how something is said—focusing on vocal tone and speaking style.

The experiment

We use a single speaker (a female, US English voice - Natalie from murf.ai) and have her deliver different sentences across a variety of tones and styles. The sentences range from neutral business jargon to casual, sentiment-laden phrases (both positive and negative). The 12 distinct tones/styles tested include: Sorrowful, Inspirational, Terrified, Furious, Newscast Casual, Conversational, Angry, Sad, Meditative, Newscast Formal, Narration, and Promo.

Research question

👉 Is it easier to classify speech into one of these 12 tones/styles using the original wav2vec2 audio embeddings, or does the “residual” version (where content has been removed) provide a cleaner signal for tone classification?

tl;dr

✅ Both linear and non-linear models perform better using the residual embeddings than the raw audio ones, where the actual spoken content appears to confuse the model.

✅ Linear and non-linear classifiers perform similarly, suggesting that the relevant information (tone/style) is well-captured in a way that can be extracted with simple linear methods.

✅ Shorter sentences provide fewer vocal cues, making tone classification harder. Residual embeddings help the most in these cases by removing content interference.

✅ With longer sentences, tone information is naturally stronger, reducing the need for content filtering—though residual embeddings still offer a slight edge.

✅ This supports the idea that removing linguistic content enhances models’ ability to focus on paralinguistic features (tone, style, speaker characteristics).

Loading the Embeddings

Let’s start by loading the data (sentences and embeddings).

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import Ridge, LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    accuracy_score, f1_score, roc_auc_score, confusion_matrix)
from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE


styles = [
    "Sorrowful", "Inspirational", "Terrified",
    "Furious", "Newscast Casual", "Conversational",
    "Angry", "Sad", "Meditative",
    "Newscast Formal", "Narration", "Promo",
]


sentences = pd.read_parquet("earnings_calls_sentences.parquet")
positive_sentences = pd.read_parquet("positive_sentences.parquet")
negative_sentences = pd.read_parquet("negative_sentences.parquet")

display(sentences.sample(5))
display(positive_sentences.sample(5))
display(negative_sentences.sample(5))

	sentence
5	For the next 100 years to come, we will contin...
36	It's not just that we recruit the best or that...
13	You see the overseas sales ratio of the in-hou...
46	Compared to the IOCs over the last five years,...
31	Not just profitable growth for ExxonMobil, but...

	sentence
25	The resilience and optimism you maintain in th...
3	That meal was incredibly delicious.
10	Your kindness and generosity are truly inspiring.
18	Everything is going so well—I feel incredibly ...
4	I'm so proud of you for achieving your goal!

	sentence
15	I couldn't be more frustrated with how things ...
8	You are the most unreliable person I've ever met.
37	It is disheartening to see that despite having...
5	You always find a way to make things worse.
1	I absolutely hate dealing with this kind of si...

df_audio_emb = pd.read_parquet("wav2vec2_audio_embeddings.parquet")
df_text_emb = pd.read_parquet("dataframe_text_embeddings.parquet")

# Convert DataFrames to NumPy arrays
X = df_text_emb.values    # shape: (n_samples, 1536)
Y = df_audio_emb.values   # shape: (n_samples, 768)

print("Text embeddings shape:", X.shape)
print("Audio embeddings shape:", Y.shape)

Text embeddings shape: (1584, 1536)
Audio embeddings shape: (1584, 768)

# Train Ridge regression model
def train_ridge_regression(X, Y, alpha=1.0):
    model = Ridge(alpha=alpha)
    model.fit(X, Y)
    return model

model = train_ridge_regression(X, Y)

# Compute residual embeddings
Y_pred = model.predict(X)
E = Y - Y_pred

df_residual_emb = pd.DataFrame(E, index=df_audio_emb.index)

df_residual_emb

	0	1	2	3	4	5	6	7	8	9	...	758	759	760	761	762	763	764	765	766	767
sent_speaker
embeddings_downloaded_audio_00_Angry.npy	0.000547	-0.005123	0.033253	-0.038299	0.013721	-0.038447	0.007130	0.023777	0.000756	-0.018350	...	-0.011541	-0.012827	0.026055	-0.032220	-0.038547	0.006204	-0.010762	0.002884	-0.038804	0.044158
embeddings_downloaded_audio_00_Conversational.npy	-0.012998	0.003234	0.010352	-0.004429	0.019514	-0.000994	0.014857	0.005752	-0.004121	-0.029278	...	-0.032924	0.002646	0.021952	0.001924	-0.037578	-0.013107	-0.020910	-0.005894	0.017660	-0.035358
embeddings_downloaded_audio_00_Furious.npy	-0.066253	0.015863	-0.018699	0.040799	-0.021040	0.018685	0.038205	-0.000754	0.012815	-0.008809	...	0.003965	0.008530	0.002404	-0.032704	0.055339	-0.023414	-0.019636	0.023885	-0.000745	-0.023359
embeddings_downloaded_audio_00_Inspirational.npy	-0.002424	-0.028308	0.000459	0.002153	-0.007286	0.022152	-0.025893	-0.025777	-0.058059	-0.007379	...	0.016637	-0.011735	0.017605	0.017890	0.033506	0.005326	-0.003464	-0.004350	-0.007208	0.026986
embeddings_downloaded_audio_00_Meditative.npy	0.001378	-0.011582	0.036844	-0.042164	-0.036087	0.010559	-0.005657	0.025860	-0.070218	0.003857	...	0.046636	0.004565	0.011343	-0.003232	-0.006497	-0.026536	0.002865	-0.012032	-0.005313	0.042849
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
embeddings_positive_text_downloaded_audio_39_Newscast Formal.npy	-0.010689	-0.001865	-0.040611	0.009453	0.047540	-0.027685	0.015540	0.012837	0.010790	0.011269	...	-0.015807	-0.010005	-0.005689	-0.000988	-0.033687	0.001929	0.008778	-0.015480	-0.008912	-0.012508
embeddings_positive_text_downloaded_audio_39_Promo.npy	-0.035135	0.026699	-0.025716	-0.008928	0.055495	0.066652	0.038595	-0.007269	-0.032083	-0.002548	...	0.044913	0.003458	0.005861	-0.050246	0.004993	0.000369	0.010904	-0.010641	-0.021717	-0.054168
embeddings_positive_text_downloaded_audio_39_Sad.npy	0.019876	0.014143	-0.087296	0.017254	0.075202	-0.049448	-0.006937	-0.039578	0.067603	-0.001711	...	-0.036482	-0.010577	-0.000425	0.051968	-0.067635	0.016389	-0.007544	-0.042562	0.022262	-0.001306
embeddings_positive_text_downloaded_audio_39_Sorrowful.npy	0.021177	-0.034878	-0.025202	0.025799	-0.004771	-0.026035	-0.018924	-0.022863	0.038629	-0.001309	...	-0.024871	0.010005	0.003231	0.040676	0.012732	-0.019660	0.016285	-0.011352	0.044231	0.067834
embeddings_positive_text_downloaded_audio_39_Terrified.npy	0.019007	-0.010457	0.005947	0.026304	-0.048532	0.004886	-0.005772	-0.014250	0.044395	0.002009	...	0.006794	0.008677	0.002136	0.020690	0.000381	-0.016183	0.008968	0.016245	0.035921	0.059694

1584 rows × 768 columns

Understanding the Embeddings

To begin, we visualize the embeddings by projecting them into a 2D space using standard dimensionality reduction techniques like PCA and t-SNE. This allows us to explore whether any clear structure emerges in these representations.

X_text = df_text_emb.values
X_audio = df_audio_emb.values
X_resid = df_residual_emb.values

# Extract labels: assuming styles are encoded in filenames
labels = pd.Series([elem.split('_')[-1].split('.npy')[0]
                    for elem in df_audio_emb.index])

def plot_pca_tsne(X, labels, title_prefix):
    """
    Apply PCA and t-SNE to reduce dimensions and visualize embeddings.
    """
    # Convert labels to categorical numeric values
    label_values, label_mapping = pd.factorize(labels)

    # Apply PCA
    pca = PCA(n_components=2)
    X_pca = pca.fit_transform(X)

    # Apply t-SNE
    tsne = TSNE(n_components=2, perplexity=5, random_state=42)
    X_tsne = tsne.fit_transform(X)

    fig, axes = plt.subplots(1, 2, figsize=(14, 6))

    # PCA Plot
    scatter = axes[0].scatter(
        X_pca[:, 0], X_pca[:, 1],
        c=label_values, cmap="tab20", alpha=0.7)
    axes[0].set_title(f"{title_prefix} - PCA Projection")
    axes[0].set_xlabel("PC1")
    axes[0].set_ylabel("PC2")

    # t-SNE Plot
    scatter = axes[1].scatter(
        X_tsne[:, 0], X_tsne[:, 1],
        c=label_values, cmap="tab20", alpha=0.7)
    axes[1].set_title(f"{title_prefix} - t-SNE Projection")
    axes[1].set_xlabel("t-SNE Dim 1")
    axes[1].set_ylabel("t-SNE Dim 2")

    # Add legend for the unique labels
    legend_labels = {i: label for i, label in enumerate(label_mapping)}
    handles = scatter.legend_elements()[0]
    labels_list = [legend_labels[i] for i in range(len(handles))]
    axes[1].legend(handles, labels_list,
                   loc="best", fontsize="small", title="Styles")

    plt.show()


    
# Run PCA & t-SNE for Text Embeddings
content_labels = pd.Series(
    ["business"] * 52 * 12 + ["negative"] * 40 * 12 + ["positive"] * 40 * 12)
plot_pca_tsne(X_text, content_labels, "Text Embeddings")

# Run PCA & t-SNE for Audio Embeddings
plot_pca_tsne(X_audio, labels, "Audio Embeddings")

# Run PCA & t-SNE for Residual Embeddings
plot_pca_tsne(X_resid, labels, "Residual Embeddings")

These visualizations confirm that residual embeddings provide clearer clustering by tone, reinforcing their usefulness in capturing vocal style.

In more details:

Text Embeddings

The first row displays the text embeddings projected onto a two-dimensional space using PCA (left) and t-SNE (right). Since these embeddings represent only the textual content and not the way sentences are spoken, the different tone/style labels do not appear distinct in the PCA projection. Instead, we only observe clusters corresponding to:

Business-related sentences (52 samples)
Positive sentiment sentences (40 samples)
Negative sentiment sentences (40 samples)

This indicates that text embeddings primarily encode content (and sentiment).

Audio Embeddings

For the raw audio embeddings, the PCA projection (center-left) does not exhibit clear structure. While a handful of outliers are present, the majority of the data points are densely packed without clear separation. This suggests that in their raw form, audio embeddings mix linguistic content and vocal tone/style, making them difficult to disentangle.

However, when applying t-SNE (center-right), we start to observe some degree of clustering by tone/style. Notably:

The “Promo” (grey) and “Furious” (orange) styles form noticeable clusters.
Other styles remain partially mixed, indicating that raw audio embeddings do encode tone information but not in a way that is easily separable.

These findings reinforce the idea that raw speech embeddings contain a blend of both linguistic and speaker-specific information, making it challenging to extract tone/style characteristics directly.

Residual Embeddings

The residual embeddings, which remove linguistic content using the elementary method presented in Disentangling Speech Embeddings, exhibit a much stronger clustering structure:

PCA projection (bottom-left) shows better-defined clusters (e.g., blue, orange, brown, grey) compared to raw audio embeddings, although the separation is not perfect.
t-SNE projection (bottom-right) reveals a remarkably clear separation of the 12 tones/styles. Unlike raw audio embeddings, the residual embeddings distinctly group sentences by how they are spoken rather than what is being said.

This result strongly suggests that removing linguistic content leaves behind a representation that is highly predictive of tone and style. In other words:

Text embeddings capture only the content of speech (what is being said).
Residual embeddings predominantly encode vocal tone/style (how it is being said).
Raw audio embeddings contain a mixture of both, making classification more difficult.

Implications

If the goal is tone/style classification, residual embeddings should be used instead of raw audio embeddings.
Residual embeddings cluster strongly by style, meaning even simple models (e.g., logistic regression) should be able to classify tone effectively.
This experiment confirms that an elementary linear disentanglement technique successfully separates tone and linguistic content in speech embeddings.

Validating the Findings with Supervised Classification

So far, our unsupervised analysis has shown that residual embeddings exhibit clearer clustering by tone/style compared to raw audio embeddings. But how well does this separation translate to practical classification performance?

To test this, we move to a supervised classification approach:

🔹 Objective: Train a classifier to predict tone/style from speech embeddings and compare performance across different representations (audio vs. residual).

🔹 Hypothesis: If residual embeddings truly capture tone/style better than raw audio embeddings, a simple classifier should achieve higher accuracy on residuals than on raw audio embeddings.

🔹 Experiment:

Train a logistic regression model and a random forest classifier on both raw audio and residual embeddings.
Compare accuracy, F1-score, and AUC-ROC across models and representations.
Evaluate whether removing linguistic content improves the model’s ability to focus on tone and style.

Let’s dive into the code and results:

# Function to train, evaluate, and return results
def evaluate_model(X, y, model, model_name):
    """Trains a model, evaluates performance, and returns metrics."""
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42)
    
    # Train model
    model.fit(X_train, y_train)
    
    # Predict
    y_pred = model.predict(X_test)
    y_proba = (
        model.predict_proba(X_test)
        if hasattr(model, "predict_proba") else None
    )
    
    # Compute Metrics
    acc = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred, average="weighted")
    roc_auc = (
        roc_auc_score(y_test, y_proba, multi_class="ovr")
        if y_proba is not None else None
    )

    y_pred = model.predict(X)
    conf_matrix = confusion_matrix(y, y_pred)
    
    return {"Model": model_name,
            "Accuracy": acc,
            "F1 Score": f1,
            "AUC-ROC": roc_auc,
            "Confusion Matrix": conf_matrix}


# Function to display confusion matrix
def plot_confusion_matrix(conf_matrix, title):
    """Plots a confusion matrix with heatmap."""
    plt.figure(figsize=(6, 5))
    sns.heatmap(conf_matrix, annot=True, fmt="d", cmap="Blues")
    plt.xlabel("Predicted Label")
    plt.ylabel("True Label")
    plt.title(title)
    plt.show()

# Prepare data for classification tasks
def prepare_classification_data(
    df_audio_emb, df_residual_emb, df_text_emb, use_case="business"):
    if use_case == "business":
        index = (
            ~(df_audio_emb.index.str.contains("positive")) &
            ~(df_audio_emb.index.str.contains("negative")))
    elif use_case == "sentimental_text":
        index = (
            (df_audio_emb.index.str.contains("positive")) |
            (df_audio_emb.index.str.contains("negative")))
    elif use_case == "short_text":
        mask = (
            df_audio_emb.index.str.extract(r'_(\d+)_')[0]
            .astype(float)
            .between(0, 19)
        )
        mask.index = df_audio_emb.index
        index = (
            (df_audio_emb.index.str.contains("positive") |
             df_audio_emb.index.str.contains("negative")) &
            mask
        ).tolist()
    elif use_case == "long_text":
        mask = (
            df_audio_emb.index.str.extract(r'_(\d+)_')[0]
            .astype(float)
            .between(20, 39)
        )
        mask.index = df_audio_emb.index
        index = (
            (df_audio_emb.index.str.contains("positive") |
             df_audio_emb.index.str.contains("negative")) &
            mask
        ).tolist()       
    
    X_audio = df_audio_emb[index]
    y_audio = [elem.split('_')[-1].split('.npy')[0]
               for elem in X_audio.index]
    
    X_resid = df_residual_emb[index]
    y_resid = [elem.split('_')[-1].split('.npy')[0]
               for elem in X_resid.index]
    
    X_text = df_text_emb[index]
    y_text = y_audio
    
    return (X_audio, y_audio,
            X_resid, y_resid,
            X_text, y_text)

X_audio, y_audio, X_resid, y_resid, X_text, y_text = (
    prepare_classification_data(
        df_audio_emb, df_residual_emb, df_text_emb,
        use_case="business"))


# Convert labels to numerical format
unique_labels = sorted(set(y_audio))
label_mapping = {label: idx for idx, label in enumerate(unique_labels)}
y_audio = np.array([label_mapping[label] for label in y_audio])
y_resid = np.array([label_mapping[label] for label in y_resid])
y_text  = np.array([label_mapping[label] for label in y_text])

# Convert DataFrames to numpy arrays
X_audio = X_audio.to_numpy()
X_resid = X_resid.to_numpy()
X_text  = X_text.to_numpy()

# Define models
log_reg = LogisticRegression(max_iter=1000, random_state=42)
random_forest = RandomForestClassifier(n_estimators=100, random_state=42)

# Evaluate models
results = []
results.append(
    evaluate_model(X_text, y_text, log_reg, "Logistic Regression (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, log_reg, "Logistic Regression (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, log_reg, "Logistic Regression (X_resid)"))
results.append(
    evaluate_model(X_text, y_text, random_forest, "Random Forest (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, random_forest, "Random Forest (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, random_forest, "Random Forest (X_resid)"))

# Convert results to DataFrame
results_df = pd.DataFrame(results).drop(columns=["Confusion Matrix"])

print("\nClassification Results:")
display(results_df)

Classification Results:

	Model	Accuracy	F1 Score	AUC-ROC
0	Logistic Regression (X_text)	0.000	0.000000	0.096726
1	Logistic Regression (X_audio)	0.888	0.889941	0.990831
2	Logistic Regression (X_resid)	0.944	0.944745	0.997317
3	Random Forest (X_text)	0.000	0.000000	0.095351
4	Random Forest (X_audio)	0.800	0.797580	0.972571
5	Random Forest (X_resid)	0.920	0.919786	0.996321

As anticipated, text embeddings do not capture tone or style information, as they solely encode linguistic content. We included them in the experiment as a control to validate this assumption, and the results confirm that tone cannot be recovered from text embeddings alone. This is clearly reflected in the classification performance shown in the bar plot below:

# Plot classification results
plt.figure(figsize=(10, 5))
results_df.set_index("Model").plot(kind="bar", figsize=(10, 5))
plt.title("Classification Performance Comparison", fontsize=14)
plt.ylabel("Score", fontsize=12)
plt.xticks(rotation=45, fontsize=10)
plt.yticks(fontsize=10)
plt.legend(loc="lower right", fontsize=10)
plt.grid(True, linestyle="--", linewidth=0.5, alpha=0.7)
plt.show()

Remark: Even with simple models like logistic regression, residual embeddings achieved over 94% accuracy. This suggests that tone information is not only well-preserved but also highly predictable once linguistic content is removed. In contrast, raw audio embeddings performed worse, likely because they mix content with tone.

X_audio, y_audio, X_resid, y_resid, X_text, y_text = (
    prepare_classification_data(
        df_audio_emb, df_residual_emb, df_text_emb,
        use_case="sentimental_text"))


# Convert labels to numerical format
unique_labels = sorted(set(y_audio))
label_mapping = {label: idx for idx, label in enumerate(unique_labels)}
y_audio = np.array([label_mapping[label] for label in y_audio])
y_resid = np.array([label_mapping[label] for label in y_resid])
y_text  = np.array([label_mapping[label] for label in y_text])

# Convert DataFrames to numpy arrays
X_audio = X_audio.to_numpy()
X_resid = X_resid.to_numpy()
X_text  = X_text.to_numpy()

# Define models
log_reg = LogisticRegression(max_iter=1000, random_state=42)
random_forest = RandomForestClassifier(n_estimators=100, random_state=42)

# Evaluate models
results = []
results.append(
    evaluate_model(X_text, y_text, log_reg, "Logistic Regression (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, log_reg, "Logistic Regression (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, log_reg, "Logistic Regression (X_resid)"))
results.append(
    evaluate_model(X_text, y_text, random_forest, "Random Forest (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, random_forest, "Random Forest (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, random_forest, "Random Forest (X_resid)"))


# Convert results to DataFrame
results_df = pd.DataFrame(results).drop(columns=["Confusion Matrix"])

print("\nClassification Results:")
display(results_df)

Classification Results:

	Model	Accuracy	F1 Score	AUC-ROC
0	Logistic Regression (X_text)	0.000000	0.000000	0.120661
1	Logistic Regression (X_audio)	0.875000	0.873449	0.978296
2	Logistic Regression (X_resid)	0.880208	0.878072	0.986610
3	Random Forest (X_text)	0.000000	0.000000	0.097720
4	Random Forest (X_audio)	0.687500	0.682460	0.943511
5	Random Forest (X_resid)	0.848958	0.846386	0.975307

Logistic Regression performs surprisingly well on raw audio embeddings, likely because some tone information is already linearly separable, as suggested by the earlier 2D projections. However, Random Forests show a much stronger improvement with residual embeddings, benefiting from the removal of linguistic content. By eliminating this noise, residual embeddings provide a cleaner representation of paralinguistic features, making them more suitable for decision-tree-based learning.

# Plot classification results
plt.figure(figsize=(10, 5))
results_df.set_index("Model").plot(kind="bar", figsize=(10, 5))
plt.title("Classification Performance Comparison", fontsize=14)
plt.ylabel("Score", fontsize=12)
plt.xticks(rotation=45, fontsize=10)
plt.yticks(fontsize=10)
plt.legend(loc="lower right", fontsize=10)
plt.grid(True, linestyle="--", linewidth=0.5, alpha=0.7)
plt.show()

Note: Before expanding the dataset with longer sentences, we initially ran this experiment using only 20 short positive and 20 short negative sentences. In that setup, logistic regression on raw audio embeddings achieved 0.71 accuracy, while residual embeddings improved performance to 0.82. This suggests that for shorter sentences, where vocal cues are more limited or entangled with linguistic content, residual embeddings provide a clearer benefit. By stripping away the linguistic information, they make it easier for the model to focus on paralinguistic features, leading to a more pronounced lift in classification accuracy.

This motivated us to examine whether longer, more expressive sentences improve classification performance overall and to assess whether the advantage of residual embeddings over raw audio embeddings diminishes as more vocal cues become available.

Short vs. Long Sentences

To explore this, we compare model performance on short versus long sentences, starting with the short sentence case.

The short sentences case:

X_audio, y_audio, X_resid, y_resid, X_text, y_text = (
    prepare_classification_data(
        df_audio_emb, df_residual_emb, df_text_emb,
        use_case="short_text"))


# Convert labels to numerical format
unique_labels = sorted(set(y_audio))
label_mapping = {label: idx for idx, label in enumerate(unique_labels)}
y_audio = np.array([label_mapping[label] for label in y_audio])
y_resid = np.array([label_mapping[label] for label in y_resid])
y_text  = np.array([label_mapping[label] for label in y_text])

# Convert DataFrames to numpy arrays
X_audio = X_audio.to_numpy()
X_resid = X_resid.to_numpy()
X_text  = X_text.to_numpy()

# Define models
log_reg = LogisticRegression(max_iter=1000, random_state=42)
random_forest = RandomForestClassifier(n_estimators=100, random_state=42)

# Evaluate models
results = []
results.append(
    evaluate_model(X_text, y_text, log_reg, "Logistic Regression (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, log_reg, "Logistic Regression (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, log_reg, "Logistic Regression (X_resid)"))
results.append(
    evaluate_model(X_text, y_text, random_forest, "Random Forest (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, random_forest, "Random Forest (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, random_forest, "Random Forest (X_resid)"))


# Convert results to DataFrame
results_df = pd.DataFrame(results).drop(columns=["Confusion Matrix"])

print("\nClassification Results:")
display(results_df)

# Plot classification results
plt.figure(figsize=(10, 5))
results_df.set_index("Model").plot(kind="bar", figsize=(10, 5))
plt.title("Classification Performance Comparison", fontsize=14)
plt.ylabel("Score", fontsize=12)
plt.xticks(rotation=45, fontsize=10)
plt.yticks(fontsize=10)
plt.legend(loc="lower right", fontsize=10)
plt.grid(True, linestyle="--", linewidth=0.5, alpha=0.7)
plt.show()

Classification Results:

	Model	Accuracy	F1 Score	AUC-ROC
0	Logistic Regression (X_text)	0.000000	0.000000	0.121146
1	Logistic Regression (X_audio)	0.708333	0.700649	0.936023
2	Logistic Regression (X_resid)	0.822917	0.817159	0.978280
3	Random Forest (X_text)	0.000000	0.000000	0.093249
4	Random Forest (X_audio)	0.552083	0.533623	0.847435
5	Random Forest (X_resid)	0.697917	0.685180	0.947323

The long sentences case:

We now turn to longer sentences, where more vocal cues are naturally present.

X_audio, y_audio, X_resid, y_resid, X_text, y_text = (
    prepare_classification_data(
        df_audio_emb, df_residual_emb, df_text_emb,
        use_case="long_text"))


# Convert labels to numerical format
unique_labels = sorted(set(y_audio))
label_mapping = {label: idx for idx, label in enumerate(unique_labels)}
y_audio = np.array([label_mapping[label] for label in y_audio])
y_resid = np.array([label_mapping[label] for label in y_resid])
y_text  = np.array([label_mapping[label] for label in y_text])

# Convert DataFrames to numpy arrays
X_audio = X_audio.to_numpy()
X_resid = X_resid.to_numpy()
X_text  = X_text.to_numpy()

# Define models
log_reg = LogisticRegression(max_iter=1000, random_state=42)
random_forest = RandomForestClassifier(n_estimators=100, random_state=42)

# Evaluate models
results = []
results.append(
    evaluate_model(X_text, y_text, log_reg, "Logistic Regression (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, log_reg, "Logistic Regression (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, log_reg, "Logistic Regression (X_resid)"))
results.append(
    evaluate_model(X_text, y_text, random_forest, "Random Forest (X_text)"))
results.append(
    evaluate_model(X_audio, y_audio, random_forest, "Random Forest (X_audio)"))
results.append(
    evaluate_model(X_resid, y_resid, random_forest, "Random Forest (X_resid)"))


# Convert results to DataFrame
results_df = pd.DataFrame(results).drop(columns=["Confusion Matrix"])

print("\nClassification Results:")
display(results_df)

# Plot classification results
plt.figure(figsize=(10, 5))
results_df.set_index("Model").plot(kind="bar", figsize=(10, 5))
plt.title("Classification Performance Comparison", fontsize=14)
plt.ylabel("Score", fontsize=12)
plt.xticks(rotation=45, fontsize=10)
plt.yticks(fontsize=10)
plt.legend(loc="lower right", fontsize=10)
plt.grid(True, linestyle="--", linewidth=0.5, alpha=0.7)
plt.show()

Classification Results:

	Model	Accuracy	F1 Score	AUC-ROC
0	Logistic Regression (X_text)	0.000000	0.000000	0.110647
1	Logistic Regression (X_audio)	0.979167	0.979601	0.996384
2	Logistic Regression (X_resid)	0.989583	0.989279	0.999267
3	Random Forest (X_text)	0.000000	0.000000	0.097023
4	Random Forest (X_audio)	0.895833	0.904243	0.989440
5	Random Forest (X_resid)	0.989583	0.989279	0.999817

This last experiment highlights an important distinction:

Short sentences provide fewer vocal cues, making it harder for models to distinguish tone from linguistic content. In raw audio embeddings, tone information is often entangled with what is being said, making classification more difficult. This is where residual embeddings provide a significant advantage, as they strip away content-related noise and preserve only vocal tone and style.
Long sentences, on the other hand, naturally contain more vocal evidence of tone, making classification easier across all models—even when using raw audio embeddings. With longer utterances, the additional speech data allows tone to emerge more distinctly, reducing the need for content filtering. Consequently, while residual embeddings still perform best, the gap between raw and residual embeddings becomes less pronounced.

In summary, residual embeddings are most beneficial when tone cues are sparse or ambiguous, such as in short speech segments. When ample vocal evidence is available (as in longer utterances), models can more easily capture tone directly from raw embeddings, making content removal less critical.

🔍 Key Takeaways

✅ Residual embeddings cluster clearly by tone, unlike raw audio embeddings, making them a more effective representation of vocal style.

✅ Removing linguistic content enhances classification accuracy, particularly in cases where tone is subtle or mixed with content.

✅ Even simple models perform well on residual embeddings, proving that the extracted paralinguistic features are well-structured and easy to classify.

✅ Residual embeddings are especially beneficial for short sentences, where tone cues are limited and harder to separate from content in raw audio embeddings.

✅ With longer sentences, tone information becomes more pronounced in raw audio embeddings, naturally improving classification performance across the board and reducing the gap between raw and residual embeddings.

✅ Residual embeddings provide the greatest advantage when tone evidence is sparse or ambiguous, reinforcing their usefulness in analyzing brief speech segments or detecting subtle vocal cues.

Discrepancy Index – A Future Research Direction

So far, we’ve demonstrated that residual embeddings are more effective at capturing tone and style, while text embeddings reflect semantic content. But what happens when textual sentiment and vocal tone don’t align?

This opens up an exciting research question: Could we quantify such discrepancies to detect sarcasm, persuasion, or even deception?

💡 The idea:

We propose a Discrepancy Index, a metric to measure mismatches between text sentiment (what is said) and vocal tone (how it is said).

🔹 Potential applications:

📢 Sarcasm detection → Positive words, but an angry tone? That’s likely sarcasm.
🕵️‍♂️ Deception & hesitation analysis → A positive tone but hesitant, uncertain speech? Possible deception or reluctance.
🎭 Politeness & persuasion → Negative wording, but spoken warmly? Maybe softening bad news.

🔬 Next steps for exploration:

Train a model to classify text sentiment (positive/negative).
Train another model to classify vocal tone from residual embeddings.
Compute a Discrepancy Score: When sentiment and tone contradict, is it meaningful?

We hypothesize that by detecting mismatches between vocal tone and textual sentiment, we can build AI models that detect sarcasm, deception, and persuasion more effectively.

For instance, in customer service, a polite “We’re happy to help” in a frustrated tone might signal dissatisfaction. In earnings calls, a neutral financial update with a hesitant tone might indicate hidden concerns.

These findings reinforce that AI can learn not just what we say, but how we say it. By exploring the Discrepancy Index for sarcasm, persuasion, and deception, we take another step toward teaching machines to ‘read the room’.

Disentangling Speech Embeddings: Removing Text Content from Audio Embeddings with Regression

Sun, 02 Feb 2025 00:00:00 +0000

Disentangling Speech Embeddings: Removing Text Content from Audio Embeddings with Regression

tl;dr: Speech embeddings encode both text content and audio-specific features (e.g., speaker identity, prosody). This blog explores how to disentangle these components by using Ridge regression to remove text content, leaving residual embeddings that focus on speaker-specific characteristics. We analyze these residual embeddings, and find that the residuals effectively isolate audio features.

Introduction
Data Preparation
Residualizing Audio Embeddings via Regression
Analyzing Embedding Similarities
Discussion and Conclusion

1. Introduction

Over the past few years, advances in deep learning—especially using transformer architectures and self-supervised learning—have revolutionized audio and speech processing. Modern models produce rich speech embeddings that encapsulate various aspects of the signal: background noise, speaker identity, prosody, and even the semantic content (i.e., what is being said).

However, if you plan to use these embeddings for downstream tasks, an important question arises: What exactly is contained in these embeddings? Are they dominated by the text content, or do they also capture pure audio characteristics? In this blog post, we walk through a notebook that tackles this question by “regressing out” the text content from the audio embeddings. We then analyze both the original and residual embeddings to see how the similarity patterns change, and whether we have isolated the speaker/audio-specific features.

Background Concepts

Before we dive into the analysis, let’s briefly define a few key concepts:

Embeddings: High-dimensional numerical representations of text or audio data, capturing semantic or speaker-specific information.
Cosine Similarity: A measure of similarity between two vectors, calculated as the cosine of the angle between them. It ranges from -1 (opposite) to 1 (identical).
Ridge Regression: A linear regression model with an L2 regularization term that prevents overfitting by penalizing large coefficients.

With these definitions in mind, let’s move on to the data preparation.

2. Data Preparation

We first load and preprocess our data. In our case, we have:

Text embeddings (shape 1536) stored in a Parquet file.
Audio embeddings (shape 768) stored as .npy files for each sentence and voice.

2.1 Loading Audio and Text Embeddings

We start by importing our libraries and reading in the embeddings.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge
from sklearn.metrics.pairwise import cosine_similarity

# Define a list of voice labels used in the recordings.
voices = [
    "alloy", "ash", "coral", "echo", "fable", "onyx", "nova", "sage", "shimmer",
]

# Load text embeddings and sentence metadata.
text_emb = pd.read_parquet("text_embeddings.parquet")
sentences = pd.read_parquet("earnings_calls_sentences.parquet")

# Process audio embeddings: compute a single (mean-pooled) vector per recording.
audio_embeddings = []
for idx in range(len(sentences)):
    for voice in voices:
        embedding = np.load(f"audio_embeddings/speech_{idx}/speech_{voice}.npy")
        # Mean pool over the time axis; adjust indexing as needed.
        single_embedding = embedding.mean(axis=1)[0].tolist()
        audio_embeddings.append([f"{idx}_{voice}"] + single_embedding)

2.2 Experimental Setup

Our experiments start with a dataset of 52 unique sentences, each spoken by 9 different voices (e.g., “alloy,” “ash,” “coral”). This results in a total of 468 audio recordings, with each sentence-voice pair represented by:

Text embeddings (dimensionality: 1536) encoding the semantic content of the sentence.
Audio embeddings (dimensionality: 768) capturing both speaker-specific characteristics and the spoken content.

This setup allows us to explore how embeddings encode information across voices and sentences and how residualizing affects these representations.

2.3 Building DataFrames

Next, we construct DataFrames for the audio embeddings. For each sentence (indexed by idx) and for each voice, we load the corresponding .npy file, compute the mean over time frames, and store the result. We then repeat the text embedding for each voice (since each sentence’s text is paired with multiple voices).

# Create a DataFrame with audio embeddings and set the index.
df_audio_emb = pd.DataFrame(audio_embeddings).set_index(0)
df_audio_emb.index.name = "sent_speaker"
display(df_audio_emb.head())

# Expand the text embeddings to match the number of audio samples.
exp_text_emb = []
for idx, row in text_emb.iterrows():
    for _ in range(len(voices)):
        exp_text_emb.append(row.values.tolist())
df_text_emb = pd.DataFrame(exp_text_emb)
display(df_text_emb.head())

	1	2	3	4	5	6	7	8	9	10	...	759	760	761	762	763	764	765	766	767	768
sent_speaker
0_alloy	-0.181595	0.044399	0.336770	0.053689	0.187663	-0.230498	0.022786	-0.104681	-0.193501	-0.176574	...	-0.056888	0.203571	0.096349	0.057072	-0.296644	-0.061813	-0.190370	0.177688	0.354035	-0.282887
0_ash	-0.136084	-0.002757	0.295544	0.086532	0.232084	-0.240367	0.033949	-0.110311	-0.150221	-0.166532	...	-0.029989	0.178773	0.108225	0.074687	-0.370774	-0.072837	-0.204104	0.226053	0.343412	-0.251210
0_coral	-0.158940	0.009858	0.358308	0.066840	0.209457	-0.220230	-0.010542	-0.112051	-0.242541	-0.176694	...	-0.013010	0.216197	0.078014	0.045757	-0.367586	-0.057285	-0.206471	0.237917	0.285875	-0.227131
0_echo	-0.147727	0.021007	0.367160	0.052192	0.219705	-0.233183	0.013074	-0.109856	-0.230388	-0.161419	...	-0.016526	0.195764	0.098858	0.127696	-0.308343	-0.074408	-0.170866	0.212565	0.339611	-0.250214
0_fable	-0.173835	0.020789	0.361774	0.049154	0.180092	-0.222793	-0.007481	-0.098986	-0.209910	-0.172046	...	-0.087433	0.161901	0.106060	0.113969	-0.340039	-0.061142	-0.185088	0.235554	0.332321	-0.299397

5 rows × 768 columns

	0	1	2	3	4	5	6	7	8	9	...	1526	1527	1528	1529	1530	1531	1532	1533	1534	1535
0	-0.017998	-0.02158	-0.007001	-0.015076	0.008172	-0.005641	-0.010977	-0.005072	-0.01783	-0.018438	...	0.004975	-0.005728	0.050969	0.001896	0.001191	-0.01849	-0.005182	-0.005217	-0.001707	-0.011934
1	-0.017998	-0.02158	-0.007001	-0.015076	0.008172	-0.005641	-0.010977	-0.005072	-0.01783	-0.018438	...	0.004975	-0.005728	0.050969	0.001896	0.001191	-0.01849	-0.005182	-0.005217	-0.001707	-0.011934
2	-0.017998	-0.02158	-0.007001	-0.015076	0.008172	-0.005641	-0.010977	-0.005072	-0.01783	-0.018438	...	0.004975	-0.005728	0.050969	0.001896	0.001191	-0.01849	-0.005182	-0.005217	-0.001707	-0.011934
3	-0.017998	-0.02158	-0.007001	-0.015076	0.008172	-0.005641	-0.010977	-0.005072	-0.01783	-0.018438	...	0.004975	-0.005728	0.050969	0.001896	0.001191	-0.01849	-0.005182	-0.005217	-0.001707	-0.011934
4	-0.017998	-0.02158	-0.007001	-0.015076	0.008172	-0.005641	-0.010977	-0.005072	-0.01783	-0.018438	...	0.004975	-0.005728	0.050969	0.001896	0.001191	-0.01849	-0.005182	-0.005217	-0.001707	-0.011934

5 rows × 1536 columns

Now we have two DataFrames:

df_text_emb of shape (n_samples, 1536), and
df_audio_emb of shape (n_samples, 768), where n_samples equals the number of sentences multiplied by the number of voices.

3. Residualizing Audio Embeddings via Regression

The key idea is to remove the text content from the audio embeddings. We assume that each audio embedding $ \mathbf{y} $ can be approximated as a linear function of the text embedding $ \mathbf{x} $:

\[\mathbf{y} = W \mathbf{x} + \mathbf{e}\]

where $W$ is the weight matrix and $ \mathbf{e} $ is the residual representing audio-specific information. We use Ridge regression to learn $W$.

3.1 Setting Up the Regression Problem

We convert our DataFrames into NumPy arrays and check their shapes.

# Convert DataFrames to NumPy arrays.
X = df_text_emb.values    # shape: (n_samples, 1536)
Y = df_audio_emb.values   # shape: (n_samples, 768)

print("Text embeddings shape:", X.shape)
print("Audio embeddings shape:", Y.shape)

Text embeddings shape: (468, 1536)
Audio embeddings shape: (468, 768)

3.2 Fitting a Ridge Regression Model

We now fit a Ridge regression model (with a chosen regularization strength) that maps text embeddings to audio embeddings.

# Use the entire dataset for training for simplicity.
X_train, Y_train = X, Y
X_test, Y_test = X, Y

# Fit the Ridge regression model.
alpha = 1.0  # Regularization strength; you may adjust this.
model = Ridge(alpha=alpha)
model.fit(X_train, Y_train)

Ridge()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

3.3 Computing Residuals

After fitting the model, we compute the predicted audio embeddings and then the residuals:

# Predict audio embeddings from text embeddings.
Y_pred = model.predict(X)

# Compute the residual: actual audio embedding minus predicted embedding.
E = Y - Y_pred
print("Shape of the residual embeddings:", E.shape)

# Optional: evaluate the model performance.
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(Y_test, model.predict(X_test))
print("Test MSE:", mse)

# Create a DataFrame for the residual embeddings.
residual_emb = pd.DataFrame(E, index=df_audio_emb.index)
display(residual_emb.head())

Shape of the residual embeddings: (468, 768)
Test MSE: 0.0011371899030286064

	0	1	2	3	4	5	6	7	8	9	...	758	759	760	761	762	763	764	765	766	767
sent_speaker
0_alloy	-0.052219	0.008099	0.030596	-0.024522	-0.031399	-0.003683	0.003598	0.029022	0.005255	-0.011372	...	-0.027316	0.008942	0.010294	-0.032126	0.044079	-0.000491	-0.006691	-0.039610	0.013437	-0.027009
0_ash	-0.006709	-0.039057	-0.010630	0.008322	0.013022	-0.013552	0.014761	0.023392	0.048536	-0.001330	...	-0.000417	-0.015856	0.022170	-0.014510	-0.030051	-0.011516	-0.020425	0.008755	0.002814	0.004668
0_coral	-0.029564	-0.026442	0.052134	-0.011371	-0.009605	0.006586	-0.029729	0.021652	-0.043785	-0.011492	...	0.016562	0.021568	-0.008041	-0.043440	-0.026863	0.004037	-0.022792	0.020619	-0.054723	0.028747
0_echo	-0.018352	-0.015293	0.060987	-0.026018	0.000643	-0.006368	-0.006114	0.023847	-0.031632	0.003783	...	0.013046	0.001135	0.012803	0.038499	0.032380	-0.013086	0.012812	-0.004733	-0.000987	0.005664
0_fable	-0.044460	-0.015510	0.055601	-0.029056	-0.038970	0.004022	-0.026668	0.034717	-0.011154	-0.006844	...	-0.057861	-0.032728	0.020005	0.024772	0.000684	0.000180	-0.001409	0.018257	-0.008277	-0.043519

5 rows × 768 columns

4. Analyzing Embedding Similarities

To understand the impact of residualizing audio embeddings, we analyze how similarity patterns change before and after regressing out the text content. Specifically, we compute pairwise similarity matrices for both the original audio embeddings and the residual embeddings. We then compare these similarity matrices by examining their coefficient distributions, visualizing their structures, and summarizing them using average similarity scores across voices. This allows us to assess whether the residual embeddings effectively isolate speaker-specific features.

4.1 Pairwise Similarity with Cosine Similarity for Each Voice

We first compute pairwise cosine similarity among the residual embeddings and visualize the distribution of similarity values for each voice using histograms.

# Compute cosine similarity on the residual embeddings.
sim = cosine_similarity(residual_emb)
df_sim = pd.DataFrame(sim, index=residual_emb.index, columns=residual_emb.index)

# Plot histograms of the upper-triangle similarity values for each voice.
plt.figure(figsize=(12, 6))
alpha = 1.0  # starting alpha for histogram transparency

for voice in voices:
    # Select sample indices that contain the current voice.
    sample = [elem for elem in df_sim.index if voice in elem]
    mat = df_sim.loc[sample, sample]
    
    # Mask lower-triangle and diagonal values.
    mask = np.triu(np.ones(mat.shape, dtype=bool), k=1)
    upper_tri_values = np.where(mask, mat, np.nan).flatten()
    
    # Filter out NaN values.
    upper_tri_values = upper_tri_values[~np.isnan(upper_tri_values)]
    
    plt.hist(upper_tri_values, bins=100, label=voice, alpha=alpha)
    alpha -= 0.10

plt.legend()
plt.title("Histogram of Residual Embedding Similarities by Voice")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

The histogram above displays the cosine similarity distributions of residual embeddings for each voice after regressing out the text content. Each colored distribution represents a specific voice (e.g., “alloy,” “ash,” “coral”), with the x-axis showing the cosine similarity values and the y-axis indicating the frequency of these values.

Key Observations:

Distinct Voice Distributions: The residual embeddings show noticeable differences in cosine similarity distributions for each voice, highlighting that voice-specific characteristics are preserved after text content is removed.

Cosine Similarity Range: Unlike the original audio embeddings, where cosine similarity values are tightly clustered due to shared text content, the residual embeddings exhibit a wider range of similarity values (from negative values to moderately high positives, up to ~0.8). This indicates reduced influence from shared text content.

To further illustrate the impact of residualization, we compare the similarity patterns of residual embeddings with those of the original audio embeddings. The next histogram visualizes the cosine similarity distributions of the original audio embeddings for each voice, offering a baseline to evaluate how much the shared text content influences these embeddings.

# Compute cosine similarity on the audio embeddings.
sim = cosine_similarity(df_audio_emb)
df_sim = pd.DataFrame(sim, index=df_audio_emb.index, columns=df_audio_emb.index)

# Plot histograms of the upper-triangle similarity values for each voice.
plt.figure(figsize=(12, 6))
alpha = 1.0  # starting alpha for histogram transparency

for voice in voices:
    # Select sample indices that contain the current voice.
    sample = [elem for elem in df_sim.index if voice in elem]
    mat = df_sim.loc[sample, sample]
    
    # Mask lower-triangle and diagonal values.
    mask = np.triu(np.ones(mat.shape, dtype=bool), k=1)
    upper_tri_values = np.where(mask, mat, np.nan).flatten()
    
    # Filter out NaN values.
    upper_tri_values = upper_tri_values[~np.isnan(upper_tri_values)]
    
    plt.hist(upper_tri_values, bins=100, label=voice, alpha=alpha)
    alpha -= 0.10

plt.legend()
plt.title("Histogram of Audio Embedding Similarities by Voice")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

The histogram above displays the cosine similarity distributions for the original audio embeddings across different voices. Each colored distribution represents a specific voice (e.g., “alloy,” “ash,” “coral”), with the x-axis showing cosine similarity values and the y-axis indicating their frequency.

Key Observations:

Overlapping Voice Distributions: The similarity distributions for different voices overlap significantly, demonstrating that shared text content dominates the original audio embeddings.

High Similarity Concentration: The cosine similarity values for the original audio embeddings are tightly clustered in a high range, predominantly between 0.88 and 0.99, highlighting the strong influence of the text content.

Unlike the residual embeddings, the original audio embeddings exhibit a narrow spread of similarity values, indicating that speaker-specific features are less prominent in the original embeddings.

4.2 Same-Voice Similarity: Audio vs. Residual

In this section, we compare the cosine similarity patterns for recordings of the same voice (e.g., “alloy” vs. “alloy”) using both the original audio embeddings and the residual embeddings. The pairwise similarity matrices and histograms illustrate distinct differences between the two representations.

Original Audio Embeddings (Same Voice)

# Compute similarity for the same voice on audio embeddings.
voice_1 = "alloy"
voice_2 = "alloy"  # Same voice comparison.

sample_1 = [elem for elem in df_audio_emb.index if voice_1 in elem]
sample_2 = [elem for elem in df_audio_emb.index if voice_2 in elem]

pairwise_sim_same_audio = cosine_similarity(
    df_audio_emb.loc[sample_1],
    df_audio_emb.loc[sample_2]
)

plt.figure(figsize=(8, 6))
plt.pcolormesh(pairwise_sim_same_audio, cmap='viridis')
plt.title("Pairwise Similarity: 'alloy' vs. 'alloy' (Audio Embeddings)")
plt.colorbar()
plt.show()

plt.figure()
plt.hist(pairwise_sim_same_audio.flatten(), bins=100)
plt.title("Histogram: 'alloy' vs. 'alloy' (Audio Embeddings)")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

print(pd.Series(pairwise_sim_same_audio.flatten()).describe())

The similarity matrix for the original audio embeddings shows consistently high values, with most cosine similarity scores tightly clustered around 0.98. This is reflected in the histogram, where the majority of values fall between 0.95 and 1.0.

count    2704.000000
mean        0.978689
std         0.013633
min         0.903744
25%         0.972316
50%         0.982532
75%         0.988346
max         1.000000
dtype: float64

Residual Embeddings (Same Voice)

# Compute similarity for the same voice on residual embeddings.
sample_1_resid = [elem for elem in residual_emb.index if voice_1 in elem]
sample_2_resid = [elem for elem in residual_emb.index if voice_2 in elem]

pairwise_sim_same_resid = cosine_similarity(
    residual_emb.loc[sample_1_resid],
    residual_emb.loc[sample_2_resid]
)

plt.figure(figsize=(8, 6))
plt.pcolormesh(pairwise_sim_same_resid, cmap='viridis')
plt.title("Pairwise Similarity: 'alloy' vs. 'alloy' (Residual Embeddings)")
plt.colorbar()
plt.show()

plt.figure()
plt.hist(pairwise_sim_same_resid.flatten(), bins=100)
plt.title("Histogram: 'alloy' vs. 'alloy' (Residual Embeddings)")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

print(pd.Series(pairwise_sim_same_resid.flatten()).describe())

In contrast, the similarity matrix for the residual embeddings reveals a broader range of cosine similarity values, spanning from approximately -0.2 to 1.0. The histogram demonstrates more variability, with similarity values distributed more evenly across the range.

count    2704.000000
mean        0.460229
std         0.187644
min        -0.210560
25%         0.338727
50%         0.468087
75%         0.593228
max         1.000000
dtype: float64

4.3 Different-Voice Similarity: Audio vs. Residual

In this section, we analyze the similarity patterns for recordings of different voices (e.g., “alloy” vs. “coral”) using both the original audio embeddings and the residual embeddings.

The results provide insights into how shared text content and speaker-specific features influence embedding similarity.

Original Audio Embeddings (Different Voices)

# Compute similarity for different voices on audio embeddings.
voice_1 = "alloy"
voice_2 = "coral"  # different voice comparison.

sample_1 = [elem for elem in df_audio_emb.index if voice_1 in elem]
sample_2 = [elem for elem in df_audio_emb.index if voice_2 in elem]

pairwise_sim_same_audio = cosine_similarity(
    df_audio_emb.loc[sample_1],
    df_audio_emb.loc[sample_2]
)

plt.figure(figsize=(8, 6))
plt.pcolormesh(pairwise_sim_same_audio, cmap='viridis')
plt.title("Pairwise Similarity: 'alloy' vs. 'coral' (Audio Embeddings)")
plt.colorbar()
plt.show()

plt.figure()
plt.hist(pairwise_sim_same_audio.flatten(), bins=100)
plt.title("Histogram: 'alloy' vs. 'coral' (Audio Embeddings)")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

print(pd.Series(pairwise_sim_same_audio.flatten()).describe())

The similarity matrix and histogram for the original audio embeddings show that even for different voices, cosine similarity values remain high, with most scores clustering between 0.96 and 0.98. This indicates that shared text content dominates the embeddings, overwhelming the differences between voices. The high similarity values reflect a lack of discrimination between different speakers when text content is present.

count    2704.000000
mean        0.968115
std         0.015492
min         0.885379
25%         0.962617
50%         0.972892
75%         0.978895
max         0.988612
dtype: float64

Residual Embeddings (Different Voices)

# Compute similarity for different voices on residual embeddings.
sample_1_resid = [elem for elem in residual_emb.index if voice_1 in elem]
sample_2_resid = [elem for elem in residual_emb.index if voice_2 in elem]

pairwise_sim_same_resid = cosine_similarity(
    residual_emb.loc[sample_1_resid],
    residual_emb.loc[sample_2_resid]
)

plt.figure(figsize=(8, 6))
plt.pcolormesh(pairwise_sim_same_resid, cmap='viridis')
plt.title("Pairwise Similarity: 'alloy' vs. 'coral' (Residual Embeddings)")
plt.colorbar()
plt.show()

plt.figure()
plt.hist(pairwise_sim_same_resid.flatten(), bins=100)
plt.title("Histogram: 'alloy' vs. 'coral' (Residual Embeddings)")
plt.xlabel("Cosine Similarity")
plt.ylabel("Frequency")
plt.show()

print(pd.Series(pairwise_sim_same_resid.flatten()).describe())

In stark contrast, the similarity matrix and histogram for the residual embeddings demonstrate a near-complete absence of similarity between different voices. The distribution of cosine similarity values spans a much wider range, centering around -0.15 with values as low as -0.5. This confirms that the residual embeddings have successfully removed the shared text content, leaving behind voice-specific characteristics. Since the embeddings now focus solely on speaker-related features, the similarity between different voices naturally drops.

count    2704.000000
mean       -0.148577
std         0.139143
min        -0.516812
25%        -0.244333
50%        -0.161543
75%        -0.065892
max         0.443491
dtype: float64

The last histogram is particularly compelling, as it illustrates the impact of residualization by showing the near-complete absence of similarity between embeddings from different voices.

4.4 Average Similarity Across Voices

Finally, in this section, we compute the average pairwise cosine similarity between all combinations of voices for both the original and residual embeddings. This provides a broader summary of the effects observed in Sections 4.2 and 4.3 by generalizing the analysis to all voice pairs. Instead of examining the entire distribution of similarities, we condense the information into average values for each voice combination.

def plot_heatmap(sim_matrix, title, xtick_labels):
    plt.figure(figsize=(8, 6))
    plt.pcolormesh(sim_matrix, cmap='viridis')
    plt.title(title)
    plt.colorbar()
    tick_positions = np.arange(sim_matrix.shape[0]) + 0.5
    plt.xticks(tick_positions, xtick_labels, rotation=90)
    plt.yticks(tick_positions, xtick_labels)
    plt.show()

def compute_avg_similarity(df_emb, voices):
    n = len(voices)
    avg_sim = np.zeros((n, n))
    for i, v1 in enumerate(voices):
        for j, v2 in enumerate(voices):
            sample_1 = [elem for elem in df_emb.index if v1 in elem]
            sample_2 = [elem for elem in df_emb.index if v2 in elem]
            sim_vals = cosine_similarity(df_emb.loc[sample_1], df_emb.loc[sample_2])
            avg_sim[i, j] = np.nanmean(sim_vals.flatten())
    return avg_sim

# Average similarity on audio embeddings.
sim_audio = compute_avg_similarity(df_audio_emb, voices)
plot_heatmap(sim_audio,
             "Average Similarity Between Voices (Audio Embeddings)",
             voices)

# Average similarity on residual embeddings.
sim_resid = compute_avg_similarity(residual_emb, voices)
plot_heatmap(sim_resid,
             "Average Similarity Between Voices (Residual Embeddings)",
             voices)

Original Audio Embeddings

The heatmap of average pairwise similarities for the original audio embeddings reveals uniformly high similarity values across all voice pairs, with most values exceeding 0.96. This aligns with the earlier observations that text content dominates the original embeddings. The lack of variability in the heatmap suggests that the original embeddings do not effectively distinguish between voices, as the textual content contributes significantly to the similarity scores.

Residual Embeddings

In contrast, the heatmap for the residual embeddings shows low or negative similarity values for different voice pairs, illustrating the success of residualization in removing text content. For the same voice combinations (diagonal elements of the heatmap), the values are noticeably higher, reflecting the preservation of speaker-specific features. The strong contrast between diagonal and off-diagonal values demonstrates that the residual embeddings capture meaningful differences between voices while retaining consistency within the same voice.

5. Discussion and Conclusion

Our analysis shows that the original audio embeddings exhibit high similarity across different voices, likely due to the dominating influence of text content (since all voices speak the same sentence). After applying Ridge regression to regress out the text component, the residual embeddings show:

High within-voice similarity: The residual still captures speaker-specific characteristics.
Low between-voice similarity: The shared text content is largely removed.

In this blog, we demonstrated how regression can effectively isolate speaker-specific features in audio embeddings by removing shared text content. However, we did not analyze in depth what characteristics remain in the residual embeddings. Do they capture pitch, tone, prosody, timbre, speaking rate, vocal intensity, or background noise? These are open questions worth further exploration. In future posts, we aim to explore these residual features in greater detail, shedding light on the nuances of what these embeddings truly represent and how they can be applied to more specialized audio analysis tasks.

Personal Reflections on NeurIPS 2024

Tue, 31 Dec 2024 00:00:00 +0000

Personal Reflections on NeurIPS 2024

Introduction

I attended NeurIPS 2024 in Vancouver with a few colleagues (Patrik, Kelvin, Lars, and Mathieu), despite the tempting weather and numerous events happening in Abu Dhabi during the winter season; Got a bad cold on the way back. This blog is an attempt to crystallize some of my personal takeaways from the conference, along with potential leads for further exploration. Naturally, it won’t be exhaustive, as I couldn’t attend every session and had other responsibilities to manage during the event. That said, I feel I gained a solid understanding of what had been happening in the fast-paced world of (academic) AI and ML over the past year.

NeurIPS (Neural Information Processing Systems), established in 1987, is one of the most prestigious conferences in artificial intelligence (AI) and machine learning (ML). Over the decades, it has grown into a global hub for researchers, practitioners, and industry leaders to share pioneering work, exchange ideas, and explore cutting-edge advancements. In 2024, NeurIPS reached a historic milestone with nearly 16,000 registrations, cementing its status as the largest academic conference in the field.

Beyond its peer-reviewed papers (talks, posters) and workshops, the event features a dynamic ecosystem of activities, including industry-sponsored events and an expansive sponsor expo (NeurIPS 2024 sponsors). This expo serves as a talent magnet, with companies—ranging from deep tech innovators to hedge funds and proprietary trading firms—eagerly recruiting PhD graduates and top researchers whose work is showcased at the conference. NeurIPS not only shapes the trajectory of AI research but also bridges academia and industry, fueling advancements across technology and finance.

Key Highlights from the Conference

I like to visualize papers on a map to navigate the main themes and find the ones I’m interested in, or even to grasp the broader trends of a conference. For example, I’ve done this for ICML 2024. However, this time, some others have already done a great job capturing the main topics of NeurIPS 2024, so no point duplicating their efforts. Here are two interactive websites that do it well:

A visualization of NeurIPS 2024 topics by Jay Alammar

NeurIPS 2024 Papers Map

Official NeurIPS visualization

NeurIPS 2024 Papers Map - Official

I somewhat prefer the ‘non-official’ version: I find it more convenient to navigate the papers by similarity.

Another interesting angle is the specialization by economic/geographic regions, which you can explore in this Sankey diagram.

Geographies-to-Topics Sankey

The conference is clearly dominated by China and the USA, which together account for about three-quarters of the papers. It’s unclear what methodology the Sankey uses—whether it attributes papers based on the authors’ university affiliations (likely) or tries to infer their nationalities from names (less likely). This is something I’ve explored before using LLMs to infer demographics from names—if you’re curious, I’ve written about it in this blog. Many papers from US universities are authored by Chinese researchers (or ABCs), highlighting China’s strong influence in AI research.

The specialization by region is also revealing. China stands out in Computer Vision and NLP and has a proportional strength in Graph Neural Networks. China’s dominance in Computer Vision suggests ongoing investments in this field, likely driven by applications in surveillance and autonomous systems. It’s notable that China is on par with the USA in RL & Robotics, but relatively weaker in “Explainable AI.” In contrast, the USA appears well-diversified across all sub-fields. Europe, while contributing fewer papers, shows a relatively stronger focus on “Explainable AI” and “Ethics & Safety.”

What’s missing? A good visualization illustrating the evolution of topics over time. I remember 10 years ago, it was all about supervised ML, structured predictions, dealing with missing and noisy labels, and active learning. Can we trace the transition from this focus on efficient labeling to the current zero-shot and few-shot paradigms? What caused some topics to fade away completely—were they trivialized by newer approaches? And do these newer paradigms fully subsume the older problems, or is it a case where they solve only certain aspects while the community shifts focus due to trends and “mode effects”?

Maybe I’ll take this up as a weekend project at some point—not super practical, but it would be fun to visualize how the field has evolved over the past 10–15 years just out of curiosity.

NeurIPS 2024 revealed some clear directions in AI/ML research. Multimodality continues to be a major focus, with researchers exploring how to integrate text, audio, video, and other modalities more effectively. Efficiency in LLMs is another prominent theme, whether through compression techniques, new architectures, or advances in inference. The role of reinforcement learning is also evolving—often intertwined with LLMs—to improve fine-tuning and enable more interactive applications. These trends highlight the ongoing balance between theoretical advancements and practical applications in the field.

Building on these broader trends, below are some of my personal takeaways and areas I found particularly interesting at NeurIPS 2024.

Personal Takeaways

Staying up-to-date in the field is a lot of work—it moves incredibly fast. It’s impossible to keep track of all the research, especially since a lot of it will turn out to be less relevant, follow wrong paths, or quickly get subsumed by superior models and approaches. Many ideas presented today will be discarded in the coming years as they are beaten by better solutions. Still, it’s a valuable exercise to try to understand and predict which pieces of work will be influential for the future of the field. I should probably commit to a fixed list for each conference—it’s too easy to ‘back trade’ and claim “I knew it!” after the fact. Feels a bit like stock picking, doesn’t it?

LLMs were, unsurprisingly, a major focus of the conference, often tying into more classic areas of ML. For instance, there were several papers exploring LLMs in combination with Reinforcement Learning (RL). If you look at the thematic map (below), you can see how LLMs subdivide into various sub-categories and connect with other branches of ML, such as privacy/security/adversarial attacks, multimodality, and alternative learning paradigms. One particularly interesting area is the investigation of using Transformers for state-space models, traditionally dominated by methods like Kalman filters.

NeurIPS 2024 Papers Map - Focus on LLMs

There were also lots of papers on applying LLMs to tackle mathematical and algorithmic problems, as well as in coding/programming tasks, like code generation and programming assistants. The industry is already seeing widespread adoption here—think Microsoft’s GitHub Copilot. The papers in this space seem to focus on pushing beyond the current capabilities or identifying and addressing corner cases that need more attention.

Another standout topic was LLM Compression Techniques, which focus on reducing the size, computational requirements, and memory footprint of large models while preserving as much of their performance as possible. The implications are obvious—lower costs, easier deployment in low-resource environments (like edge devices), and improved energy efficiency, which aligns with broader goals of sustainability. This area is crucial given how resource-hungry LLMs can be (GPUs/TPUs don’t come cheap, and energy costs are a concern for large-scale adoption). The emphasis on this topic shows the growing demand for solutions that make LLMs more accessible and environmentally sustainable.

Interestingly, the focus on efficient inference was reflected in the presence of several companies showcasing fast inference hardware, like Cerebras, D-Matrix, and SambaNova. Surprisingly, Groq wasn’t there—or at least I didn’t spot them.

NeurIPS isn’t typically the go-to conference for deep dives into knowledge graphs (KGs), but there were still some interesting contributions, particularly in how LLMs and KGs interact. For example, papers like KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge explore how LLMs can integrate with KGs, enriching their ability to reason over structured and unstructured data.

LLMs themselves can be thought of as fuzzy lookup tables or compressed knowledge bases, as they inherently store and compress vast amounts of the world knowledge they’re exposed to during training (cf. Hochreiter’s slide below). However, their “fuzziness” makes them less precise than KGs when it comes to exact reasoning, which is why the two technologies often complement each other.

LLMs are like databases - Sepp Hochreiter

Another promising direction is the use of LLMs to build, maintain, and enhance ontologies or KGs—tasks that have traditionally been labor-intensive and require substantial manual effort. LLMs can help automate ontology learning, as seen in End-to-End Ontology Learning with Large Language Models, or assist with challenges like entity alignment despite noisy data (Entity Alignment with Noisy Annotations from Large Language Models).

The synergy between KGs and LLMs is notable because it bridges the precision of symbolic representations with the adaptability and generalization of neural models. Whether fine-tuning LLMs on KGs, using LLMs to extract and structure knowledge, or leveraging KGs to ground LLMs for better factual accuracy, this interplay is likely to grow into a critical subfield of AI research.

Another relatively significant chunk of papers at the conference focused on the theme of transformers for state space models (SSMs). The potential applications are broad, spanning time-series forecasting, control systems (e.g., dynamic modeling for robots or autonomous vehicles), and signal processing. This stream of research aims to merge the strengths of traditional SSMs, which excel at handling latent states and noise in dynamical systems, with transformers, known for their ability to model long-range dependencies and complex, non-linear relationships.

The idea is intriguing—by blending these paradigms, researchers hope to create models that are both expressive and interpretable, leveraging the strengths of both approaches. While transformers have revolutionized sequence modeling, they face challenges like high resource demands and limited interpretability, which this hybrid approach could potentially address.

Transformers for state space models

Mamba: A New Competitor to Transformers?

One particularly interesting development in this space is the emergence of Mamba, a relatively recent architecture introduced on December 1, 2023. Despite its youth, Mamba already has more than two dozen papers accepted at NeurIPS 2024 leveraging the architecture, suggesting it’s gaining traction rapidly. Could Mamba-like models represent the next trend in sequence modeling? It’s too early to tell, but the pace of adoption at this year’s NeurIPS hints at strong interest.

Mamba is positioned as an alternative to transformers, specifically designed to improve efficiency and interpretability for tasks where sequence modeling intersects with dynamical systems. This architecture seeks to address some of the computational and scalability challenges of transformers while retaining their ability to handle complex dependencies.

While I’m not deeply familiar with this literature yet, it’s clear that Mamba and transformer-SSM hybrids are generating a lot of excitement. There seems to be real potential here, not just for theoretical advancements but for practical applications in areas like economic and financial time-series forecasting or autonomous systems. I should probably find the time to catch up on this literature, as it feels like a promising development. Will Mamba-like models carve out a niche alongside or even rival transformers?

I’ve bookmarked a handful of papers that caught my attention and seem relevant for understanding Mamba and its applications. I might read them later… These include:

LLM-based agents are an exciting and rapidly growing trend in research, with significant promise for industrial applications. These agents leverage the capabilities of large language models to perform complex, autonomous tasks across various domains. The flexibility and generalization capabilities of LLMs make them a strong candidate for automating workflows, enabling better decision-making, and even simulating human-like collaboration. Below are some key themes and papers on this topic that stood out:

LLMs as Urban Agents:

Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generation: This paper explores LLM-based agents in the context of urban mobility, treating them as “virtual residents” to generate and simulate personal mobility patterns. These applications hold potential for urban planning, traffic management, and optimizing public transport systems. While similar use cases have been studied in agent-based modeling literature, they often rely on simpler, non-LLM-based agents. The novelty here lies in incorporating LLM-based agents into the modeling process, enabling more sophisticated and adaptable simulations.

Coding and Bug-Fixing Agents

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents: Investigates the use of LLM-based agents for testing and validating software bug fixes, providing insights into how autonomous agents can improve software reliability.
MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue ReSolution: MAGIS introduces a team of four specialized agents: a Manager, Repository Custodian, Developer, and Quality Assurance Engineer, each tailored for different aspects of software evolution. This multi-agent setup demonstrates how LLMs can coordinate complex, collaborative workflows in software development.

Data Science and Workflow Automation

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?: This paper provides a sobering look at the limitations of current multimodal agents, particularly in automating data science workflows. According to the authors, these agents still struggle with GUI interactions, highlighting the challenges of translating high-level language capabilities into precise, context-sensitive actions.
Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning: Focuses on optimizing data for instruction tuning through LLM-based agents, demonstrating their utility in automating large-scale data preprocessing and curation tasks.

Multi-Agent Environments

AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Making: Explores the design of adaptive environments where multiple LLM-based agents interact with social structures to solve decision-making problems. The emphasis on social dynamics and collaboration adds a layer of complexity, mimicking real-world team interactions.

Financial Decision-Making

FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making: FinCon employs a hierarchical structure with Manager and Analyst agents collaborating through natural language interactions. The system generalizes across financial tasks, including single-stock trading and portfolio management, showcasing the potential of LLM-based agents in quantitative finance.

Manager and Analyst agents - Automating the pod model?

Before LLM-based agents took center stage, the main “agents” discussed at ML conferences were those built on Reinforcement Learning (RL). As can be seen on the conference papers map, these two sub-fields are increasingly intertwined. Many RL agents now incorporate LLMs either as a core component or as part of their input pipeline. LLMs provide these agents with enhanced natural language understanding, which allows them to process unstructured or complex instructions, interact with users more naturally, and even reason in ways that traditional RL agents struggle with.

Interestingly, this interaction works in both directions. RL is not just a consumer of LLM capabilities—it’s also a tool for improving or aligning LLMs. Techniques like Reinforcement Learning from Human Feedback (RLHF) have been key to fine-tuning LLMs for tasks that require alignment with human values or preferences, such as making conversational agents more polite or factual.

The multimodal large language model (LLM) space is a growing area of research, evident from a distinct cluster of papers in the NeurIPS papers map. These models sit at the intersection of traditional LLMs (text-based) and systems designed for text-to-image and text-to-video generation. Among the various approaches, researchers are focusing on both speed improvements and expanding multimodal capabilities, including audio, visual, and temporal understanding.

Some papers tackle audio-visual learning, addressing how to combine spoken language with visual cues for tasks like captioning, transcription, and video understanding. There is also interest in audio-only tasks, though these remain a niche area within NeurIPS, as ICASSP (International Conference on Acoustics, Speech, and Signal Processing) is the more dedicated venue for such research.

Video-based LLMs (VLMs) are emerging as a focal point. These models aim to integrate textual descriptions with video inputs to build temporal understanding. However, this remains a challenge due to the complexity of aligning text with dynamic, multi-frame video content. Temporal coherence—understanding events in sequence—is an area where significant progress is needed.

Generating videos remains one of the hardest tasks in multimodal modeling. Many systems struggle to respect common physics, such as gravity or object permanence, leading to outputs that break realism. Despite these flaws, improvements in video-language models (VLMs) signal progress, and this area remains an active focus of the AI/ML community.

Modeling text, audio, and video together remains a complex task due to the differences in data structure, scale, and temporal characteristics of these modalities. While progress is being made, truly unified multimodal models are still at an early stage. Advancements in architecture design, pretraining strategies, and computational efficiency will likely be needed to handle these challenges effectively.

Multimodal data remains an underutilized source of knowledge. Current LLMs are thought to have covered much of the publicly available text-based data, and future improvements in AI understanding may depend on incorporating other modalities:

Audio: Podcasts and expert discussions capture nuanced information and perspectives not commonly available in written form.
Video: Videos contain knowledge about the physical world, including motion, spatial relationships, and common-sense physics, which text-based models cannot fully grasp.

If models like VLMs improve their ability to integrate and analyze these modalities, they could lead to better internal representations and broader applications in areas like robotics, education, and healthcare.

While multimodal LLMs are growing, diffusion models remain a prominent area of research, particularly in image generation. These models dominate this domain due to their ability to generate high-quality, high-resolution images.

A notable trend is the rise of Diffusion Transformers, which combine diffusion processes with the attention-based capabilities of transformers. These models are proving to be highly effective for generating detailed, structured images. However, new challengers like Diffusion Mamba are emerging, potentially offering a more efficient alternative to the transformer-based architecture.

Despite the ongoing dominance of diffusion models in image generation, GPT-like autoregressive (AR) models are now surpassing diffusion transformers in certain settings. Notably, the paper Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction received the NeurIPS Best Paper Award for its groundbreaking work. This research highlights how AR models can achieve state-of-the-art results in image generation by leveraging scalability and precision in next-pixel or next-scale predictions.

It has been a while—probably about five years—since I last followed the developments in Graph Neural Networks (GNNs), and NeurIPS 2024 gave me a useful update on how the field has evolved. GNNs are no longer confined to traditional message-passing paradigms; instead, the field has diversified into new architectures, scalability methods, and integration with other models like LLMs.

As a quick refresher, GNNs aim to predict various properties or relationships in graphs, tackling tasks like node classification, link prediction, graph classification, graph regression, dynamic graph prediction, and even graph generation. Graphs—or networks, as they’re often called in social sciences—are everywhere in science and industry, with applications ranging from fraud detection and drug discovery to marketing and social network analysis.

One interesting direction is the development of “universal” GNNs—models designed to generalize across different types of graphs and tasks. This feels a bit like the “GPT moment” in NLP, where the goal is to create a general-purpose model. Papers like ProG: A Graph Prompt Learning Benchmark and RAGraph: A General Retrieval-Augmented Graph Learning Framework explore this idea. That said, the concept still seems far from realization. Graph data can vary a lot—from molecular graphs to social networks—so building a one-size-fits-all solution is a tough challenge.

To dig deeper, I bookmarked a few more papers that caught my attention during the conference. These papers span foundational tasks, interpretability, and novel methods in GNNs:

Foundational Tasks: Mixture of Link Predictors on Graphs
GNNs and LLMs: LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
Interpretability: The Intelligible and Effective Graph Neural Additive Networks

And, finally, a potentially useful trick for the practitioner: Probabilistic Graph Rewiring via Virtual Nodes.

Overall, there is about five years of research to catch up on. It will be interesting to see whether graphs or time-series models will achieve their “universal” moment first—if such a thing is even possible for either field.

GNNs and causality appear close in the NeurIPS 2024 papers map, as both fields rely heavily on graphs—directed acyclic graphs (DAGs) in the case of causality. GNNs focus on modeling complex relationships and interactions in networks, often with a prediction angle, whereas causality shifts the focus to uncovering and quantifying cause-and-effect relationships, leaning more towards explanation.

GNNs and Causal AI

Within the causality cluster, there’s a clear split: Causal Discovery and Representation is positioned closer to GNNs, while Causal Inference and Experimental Design sits further away. This reflects the focus of each area—causal discovery often involves graph-based methods to uncover relationships, while causal inference is more about estimating effects once a causal structure is known.

Causal Discovery vs. Causal Inference - NeuRIPS 2024 Tutorial

Causal discovery tries to answer the question, “What causes what?” It works on identifying relationships between variables, often using observational data, and aims to represent these as a DAG or similar graphical structure. Causal inference, on the other hand, answers, “How much does X affect Y?” It focuses on quantifying effects under interventions and testing the robustness of those conclusions. Together, they form a complementary loop: discovery generates hypotheses about causality, and inference tests and refines them.

While I don’t actively use causal discovery methods in my work, I’ve picked up some high-level insights from colleagues who specialize in this area. For domains where interventions are difficult or impossible, causal discovery becomes the primary focus. However, finding robust results in practice is incredibly challenging. It would be useful to see a systematic review ranking the applications of causal discovery methods from “easy” to “hard/not yet applicable”. I don’t have the bandwidth to experiment widely, but clarifying where the field stands in terms of practical applications would be helpful.

Causal discovery is an exciting area because of its potential to improve both interpretability and robustness in AI systems. Its applications span a wide range of fields, from healthcare to economics. A few papers and tools in this space caught my attention:

Causal-learn: Causal Discovery in Python: A Python library designed to make causal discovery more accessible.
Causal Dependence Plots: This approach visualizes how an outcome depends on changes in a predictor, along with the subsequent causal changes in other predictors.
Do causal predictors generalize better to new domains? Predictors using all available features, regardless of causality, have better in-domain and out-of-domain accuracy than predictors using causal features (!).
Discovery of the Hidden World with Large Language Models A paper showing how LLMs and causal discovery methods can complement each other in factor proposals.
Causal vs. Anticausal merging of predictors Authors show that if we observe all bivariate distributions, the CMAXENT solution reduces to a logistic regression in the causal direction and Linear Discriminant Analysis (LDA) in the anticausal direction.

Overall, while the theoretical tools in causal discovery are advancing, applying them effectively in real-world scenarios seems to remain a big challenge. It’s an area worth watching.

Moving from causality to another application-driven area, healthcare stood out as a distinct cluster on the NeurIPS 2024 map, positioned between ‘Large Language Models’ and ‘Multimodal Large Language Models,’ with 14 papers focused on LLM applications in medicine.

AI for Healthcare

Healthcare has long been a major user of applied causality, particularly through RCTs (randomized controlled trials) and causal inference methods. What’s new in these NeurIPS contributions is the focus on leveraging LLMs and introducing novel benchmarks. These papers primarily explore how LLMs can assist in medical tasks while starting to touch on some notion of causality—especially in understanding and reasoning about the decisions made by these models in high-stakes environments like medicine.

Interestingly, this highlights a potential convergence: while healthcare has historically used causality to ensure robust and interpretable outcomes, the next step may involve integrating causality into LLMs themselves. This is an active area of research, and it was even explored in a NeurIPS 2024 tutorial (picture below).

LLMs and Causality - NeuRIPS 2024 Tutorial

While I don’t work in this area (data science for healthcare), it’s of interest to ADIA Lab—an independent laboratory dedicated to data and computational sciences. Exploring these works could inspire research collaborations or yield practical insights.

What makes this collection particularly interesting is its focus on practical constraints and high-stakes decision-making, hallmarks of the medical field. Strict data privacy regulations, the need for interpretable models, and the severe consequences of errors set a high bar for performance. These challenges force researchers to design rigorous benchmarks and address real-world usability, offering valuable lessons for other domains where decisions carry material consequences.

Across these papers, several common themes emerge: improving the interpretability and reliability of LLMs, designing benchmarks tailored to medical tasks, and addressing safety, bias, and real-world usability. Here’s a brief overview of the contributions:

Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE A multi-task model integrating six medical tasks, from question answering to image classification.
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications The first pediatric-specific LLM for Chinese medical applications.
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making A multi-agent framework for collaborative decision-making, excelling in multi-modal reasoning tasks.
MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning A system designed for question-asking to improve reliability in clinical reasoning.
Instruction Tuning Large Language Models to Understand Electronic Health Records Focuses on improving LLM performance in answering EHR-related patient queries.
DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models A dataset for evaluating diagnostic reasoning and interpretability compared to human doctors.
A teacher-teacher framework for clinical language representation learning A framework for aligning pre-existing models to better represent clinical notes.
MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey A benchmark covering four stages of a patient’s clinical journey, with tasks across 12 datasets.
MEDCALC-BENCH: Evaluating Large Language Models for Medical Calculations Focused on evaluating LLMs’ ability to perform medical calculations, highlighting their current limitations in clinical settings.
EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries A benchmark built on discharge summaries, evaluating LLM performance in answering patient-specific questions.
EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records A framework for verifying consistency in EHRs.
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models A benchmark dataset measuring medical safety in LLMs, emphasizing fine-tuning for improved performance.
SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark A benchmark of diverse natural language queries for medical tasks.
Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias Highlights biases in LLMs’ disease prevalence representation across demographics.

These papers demonstrate the ongoing efforts to create safe, interpretable, and practical AI systems for healthcare. Even for those outside the domain, the rigor and high standards required for medical applications can serve as a model for developing AI in other critical areas.

Climate science is another area of focus for ADIA Lab, which has a natural interest in understanding long-term environmental trends and their impacts.

What’s the difference between climate and weather? In simple terms, weather describes the short-term atmospheric conditions in a specific place—think hours to days. Climate, on the other hand, is the average of weather conditions over a much longer period, typically decades or centuries. While weather forecasts tell you if it will rain tomorrow, climate models help predict long-term changes like global warming or shifts in precipitation patterns.

Climate papers:

Validating Climate Models with Spherical Convolutional Wasserstein Distance Introduces a method to measure differences between climate models and reanalysis data, ensuring better model validation.
ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction Proposes a benchmark to extend predictability from weeks to months, filling a crucial gap in climate forecasting.
Probabilistic Emulation of a Global Climate Model with Spherical DYffusion Explores whether the success of data-driven methods in weather forecasting can extend to the complexities of climate modeling.

Weather papers:

Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Models Introduces LM-WEATHER, a model outperforming state-of-the-art results across tasks like forecasting and imputation.
Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling Combines physics for large-scale weather patterns with AI for adaptive corrections, achieving state-of-the-art performance at multiple timescales.
Probabilistic Weather Forecasting with Hierarchical Graph Neural Networks Combines latent-variable modeling with graph-based forecasting for more flexible and reliable probabilistic weather predictions.
Scaling transformer neural networks for skillful and reliable medium-range weather forecasting A transformer-based model delivering state-of-the-art performance, especially beyond 7-day forecasts, while using significantly less training data and compute.

These papers highlight the growing integration of physics-informed AI and advanced deep learning methods in tackling the challenges of both weather and climate prediction.

There were several other topics at NeurIPS 2024 that I didn’t dive into, such as molecular and quantum computing, autonomous driving, and low-level optimization techniques for ML models. While these areas are undoubtedly interesting and important, they are a bit far from my current interests and expertise.

What’s next?

Looking back at NeurIPS 2024, there are plenty of ideas to dig into and explore further. Here’s what I plan to focus on next:

Reach out to some of the authors of papers that caught my attention and invite them to present their work at the Abu Dhabi Machine Learning Meetups (or the Hong Kong ones for HK-based researchers). These sessions are a great way to learn directly from the researchers and get a better understanding of their work in a more pedagogical setting.
Try out a couple of GitHub repos associated with papers I bookmarked. If anything interesting comes out of playing with the implementations—whether insights or practical tweaks—I’ll blog about it. Not necessarily aiming for something groundbreaking, but more as a way to deepen my understanding.
Work through the stack of papers I’ve bookmarked from the conference. I’m particularly interested in topics like multimodal LLMs, Diffusion Transformers, and the Mamba architecture—these seem like promising directions worth exploring further.
Start keeping tabs on the emerging themes and discussions as they develop toward ICML 2025. It’s always better to go into a conference with a clear idea of what to prioritize.
Spend a bit more time understanding areas I’m less familiar with, like video-language models or the integration of text, audio, and video. These seem like foundational challenges, and it’s probably the next big frontier in AI.

Plenty to keep busy with. These conferences always leave me with a mix of curiosity and a bit of FOMO—there’s so much happening that you can’t possibly cover it all…

For those in Abu Dhabi (or Hong Kong), consider joining one of our upcoming Machine Learning Meetups. We’ll be discussing some of these trends in more depth, and it’s a great chance to learn from each other.

Wrapping up

NeurIPS 2024 brought a wide range of progress in AI/ML, from foundational work on multimodal models to practical applications in agents and generative systems. As always, it’s impossible to cover everything, but the conference gave me a good sense of where the field is heading and plenty of ideas to dig into. It’ll be interesting to see how these trends develop over the next year.

Are there any under-the-radar papers or trends I missed? Let me know!

What’s in a Name? AI Meets the Sociology of Naming

Sun, 17 Nov 2024 00:00:00 +0000

What’s in a Name? AI Meets the Sociology of Naming

Introduction: Context of this blog

My first encounter with this topic occurred while working on clustering financial time series to identify relative value opportunities in credit default swaps and develop alternative risk models. Around the same time, I was exploring open data sources and came across yearly datasets released by INSEE, which detailed the number of first names given to people born each year in France. These datasets allowed for the reconstruction of time series over the past century, enabling analyses such as measuring correlations between first names, clustering based on attribution patterns, and identifying similarities and trends.

This became an excellent use case for testing the flexibility of the DataGrapple platform, a tool designed to visualize CDS spread time series and hierarchical clustering from my research. Collaborating with Philippe Donnat and his team at the time, we integrated the first-name time series and hierarchical clusters into the platform, showcasing its visualization capabilities. Alongside this technical work, I developed an interest in the sociological aspects of naming and purchased Baptiste Coulmont’s book, which delves into the sociology of first names in France.

Fast forward 10 years, I encountered a tangentially related problem while working with my colleagues Abdulla and Khaled, who are pursuing PhDs in Data Science. As part of their research, they explored a dataset containing names (of both individuals and companies) but with few other accompanying features. Recognizing that names often encapsulate rich information about the entity they represent (see the field of onomastics), we explored the idea of using Large Language Models (LLMs) to extract characteristics from names and enrich the original dataset.

Although the research is still in its early stages, Abdulla and Khaled have already summarized some of their findings in two preprints:

Interestingly, these same tools have the potential to advance historical research, particularly in fields like prosopography, where patterns across groups of individuals—such as their names, origins, and societal roles—reveal broader historical dynamics. To better appreciate the possibilities for combining AI with such research, it’s worth delving into the historical development of socio-onomastics, a discipline that has long sought to understand the cultural and social significance of names.

Introduction: Brief history of socio-onomastics

The sociology of first names and surnames is not a new subject. Studies on the topic date back as early as 1681, such as the Traité de l’origine des noms et des surnoms (apparently available on Amazon).

This field, known as socio-onomastics, emerged as a sub-discipline of sociology in the late 1970s.

In France, sociologist Baptiste Coulmont has studied this topic for over two decades. He authored Sociologie des prénoms (2011, with a third edition published in 2022) and hosts a website illustrating the relationship between first names and results on the French national exam.

In France, a person’s full name (first name + last name) can reveal a surprising amount of information, such as gender, year of birth, social status, and even geographic origin—often pinpointing the region and sometimes even the specific village. This phenomenon is particularly pronounced in France, which boasts more than 900,000 different surnames). However, similar patterns can be observed to varying degrees in other parts of the world.

Key takeaways from socio-onomastics research

Cultural Influence: Media, religion, and globalization shape naming trends.
- “Kevin” in France became popular in the 1990s due to American pop culture, including actors like Kevin Costner.
- “Chloé” gained traction in part due to its association with the French luxury fashion brand.
- “Marie” remained dominant in earlier centuries because of its connection to the Virgin Mary and Catholic traditions.
Social Stratification: Certain names are linked to class or community perceptions.
- “Éléonore” or “Hugo” are often associated with upper-class or more academically inclined families in France.
- “Jordan” or “Dylan” are sometimes linked to working-class families, reflecting aspirational naming trends inspired by celebrities or sports figures.
- “Kevin” became stigmatized over time in France, being stereotypically linked to lower social classes.
Regional Identity: Names can pinpoint geographic and cultural origins.
- “Jean-Baptiste” is more common in rural, Catholic regions of France, reflecting traditional values.
- In Brittany, names like “Gwenaël” or “Maël” are tied to Celtic heritage and local identity.
- Surnames like “Le Goff”: This name is prevalent in Brittany, a region in western France, and reflects the area’s Celtic linguistic heritage.
- Surnames like “Barthez”: Common in Occitan-speaking regions such as southern France, particularly in Languedoc.
- Surnames like “Noël”: Frequently found in northeastern France, particularly in Lorraine, where it often reflects historical Catholic traditions associated with Christmas.
- Surnames like “Dubois”: While common across France, it historically originates from regions with significant woodland areas, such as Normandy or Picardy, as it means “of the woods.”
- Surnames like “Schmitt”: This surname is common in Alsace and reflects the region’s Germanic influence, as Alsace has historically shifted between French and German control. “Schmitt” is derived from the German word Schmied, meaning “smith.”
Generational Shifts: Popularity cycles reflect evolving societal values.
- “Jacques,” once a staple name for boys in older generations, has largely fallen out of favor in modern France.
- “Emma” surged in the 2000s as a modern, elegant alternative to older names like “Marie.”
- Names like “Théo” and “Léo” are increasingly popular among younger parents, reflecting contemporary naming preferences.
Ethnic Identity: Names often preserve cultural roots within diverse societies.
- “Mohamed” remains a popular choice among families of North African origin in France, reflecting cultural and religious ties.
- “Giuseppe” or “Enzo” among Italian immigrant communities highlights the retention of cultural heritage.
- “Wei” or “Xiao” are common names among Chinese immigrants, signaling strong ties to their linguistic and cultural origins.

Case Study: Gautier

What does ChatGPT 4o know about “Gautier”?

Prompt:

What can you tell about the first name “Gautier” in this context?

ChatGPT 4o:

The first name Gautier offers an interesting lens for socio-onomastics in the French context, as it reflects cultural, historical, and social influences:

Historical and Cultural Context

Origins: The name Gautier is derived from the Germanic Waldhari (wald- meaning “rule” and -hari meaning “army”). It has deep medieval roots and is closely tied to the Frankish heritage in France, making it a historically noble or prestigious name.
Popularity: Gautier has experienced revivals during different periods. In the 19th and early 20th centuries, it was relatively rare, often associated with historical or literary connotations, such as the poet Théophile Gautier. Its popularity increased in the late 20th century as part of a broader trend of reviving older, traditional French names.

Social Class: In modern times, Gautier is often associated with upper-middle-class families or parents seeking a name that blends tradition and sophistication.
Spelling Variants: The choice of spelling can also reveal social nuances:
- Gautier (traditional and French spelling) tends to appear among families with an appreciation for history or classic French culture.
- Gauthier (with an “h”) is equally traditional but sometimes perceived as slightly more formal or tied to surnames that have become first names.

Regional Relevance

Geographical Association: The name may be more common in regions of France with strong medieval legacies or historical ties to Frankish culture, such as the Loire Valley or Burgundy. However, it is not strongly tied to any one region, making it relatively “neutral” geographically.

Abstract Pattern

Generational Shift: Like many older French names, Gautier has benefited from a revival trend among parents looking for names that are both distinctive and rooted in French tradition. This reflects broader societal interest in blending heritage with modern identity.

In summary, Gautier serves as a good example of a name that is culturally rich, socially nuanced, and representative of broader trends in French naming practices.

Prompt:

If you were to provide an educated guess, what is the most likely year of birth of “Gautier”?

ChatGPT 4o:

Based on naming trends in France, an educated guess for the most likely year of birth for someone named Gautier would fall in the 1990s to early 2000s. Here’s why:

Revival of Traditional Names: During the late 20th century, there was a resurgence of interest in older, classic French names, including medieval and aristocratic names like Gautier. This aligns with a broader cultural trend toward reviving names with historical significance.
Distinctive yet Timeless Appeal: Names like Gautier became attractive to parents who wanted something traditional yet distinctive, without being overly old-fashioned.
Peak Popularity Data: While exact statistics for Gautier may not always be available, names in the same category (like Arthur or Thibault) typically saw a rise in the 1990s and early 2000s.

If we were to pick a single most likely year, 1995 would be a reasonable estimate, as this was a peak period for names of this type in France. However, it would still be commonly used throughout the decade surrounding this time.

Note: Data sourced from INSEE, showing the annual number of individuals named ‘Gautier’ born in France:

Prompt:

If you were to provide an educated guess, what is the most likely average grade for “Gautier” at the French Baccalauréat (French national exam)?

ChatGPT 4o:

Analyzing the performance of students named “Gautier” in the French Baccalauréat reveals notable trends:

High Success Rate: In 2023, 100% of candidates named “Gautier” passed the Baccalauréat.
Mentions Très Bien: The name “Gautier” is among those whose bearers frequently achieve the highest honors, indicating strong academic performance.

These patterns suggest that students named “Gautier” often excel in their studies, achieving commendable results in the Baccalauréat.

Note: Data sourced from Baptiste Coulmont’s website, illustrating performance of ‘Gautier’ vs. the national average results in the French Baccalauréat:

Let’s now combine the first name “Gautier” with the surname “Marti” to analyze the sociological insights a full name can reveal.

Prompt:

Based solely on the combination of the first name “Gautier” and the surname “Marti,” and without referencing any specific individual, what can be inferred about the person’s likely cultural background, regional origin, and other sociological aspects?

ChatGPT 4o:

Analysis:

The name “Gautier Marti” offers insights into cultural and regional origins:

First Name: Gautier
- Origin: “Gautier” is the French form of “Walter,” derived from the Germanic elements “wald” (rule) and “hari” (army), meaning “ruler of the army.”
- Usage: Predominantly used in France, “Gautier” is a masculine name with historical significance, often associated with medieval French nobility.
Surname: Marti
- Origin: “Marti” is a surname found in various cultures:
  - Catalan: Derived from the personal name “Martí,” a form of “Martin,” meaning “of Mars” or “warlike.”
  - Swiss German and South German: A diminutive of “Martin.”
  - Italian: Possibly from the medieval Greek “Martios,” meaning “March,” indicating a connection to the month.
- Usage: Common in Catalonia (Spain), parts of France, Switzerland, and Italy.

Note: Interestingly, I briefly considered the army as a potential career path, and went through the competitive exams for officer positions in the French Air Force and the French Navy.

Sociological Inferences:

Cultural Background: The combination suggests a French cultural background, given the French first name “Gautier” and the presence of the surname “Marti” in southern France.
Regional Origin: The surname “Marti” is prevalent in southern France, particularly in regions like Languedoc and Provence.
Historical Context: The name “Gautier” has medieval French associations, while “Marti” may indicate Catalan or southern French heritage.

“Gautier Marti” likely denotes a male individual of French nationality, possibly from southern France, with a name reflecting both medieval French and regional influences.

Overall Hypothesis:

Gautier Marti is likely a male individual of French nationality, born in the 1990s, with a mixed northern and southern French heritage. The name suggests an upper-middle-class background and possibly a family with a strong sense of cultural or historical appreciation.

Note: ChatGPT 4o is accurate in its assessment of me, based on first name + last name only!

Modern Tools and AI in Onomastics

As demonstrated in the anecdotal “Gautier” case study, Large Language Models (LLMs) can infer a surprising amount of background information about a person based solely on their first name—and even more accurately when a last name is included. These models, trained on vast datasets, excel at identifying cultural, historical, and regional naming patterns, providing insights that were once the exclusive domain of sociological expertise.

Together with Khaled and Mathieu, we aimed to provide a more comprehensive evaluation of LLMs’ capabilities, extending beyond the relatively “easy” case of French nationals. Our focus was on assessing performance across datasets with different geographic emphases: an American-focused dataset (Florida Voters Registration), a global dataset (Wikipedia People, albeit with a strong American bias), and an Asia-focused dataset (the SFC register of Hong Kong finance professionals, with a strong emphasis on Chinese names).

The results are detailed in Khaled’s paper, Enriching Datasets with Demographics through Large Language Models: What’s in a Name?. In summary, we found that these models are highly accurate in predicting gender, ethnicity, and nationality but face greater challenges when estimating age or date of birth. Some countries lack clear trends in naming patterns over time. For instance, in countries where first names are tied to religion, names tend to be timeless and remain in consistent use across generations. In others, like China, unisex names present difficulties in determining gender. Additionally, some Asian countries have a very limited pool of surnames—for example, South Korea, where Kim accounts for 21.5% of the population, or Vietnam, where Nguyễn is used by 39% of the population, reflecting historical practices of adopting the family names of prominent rulers.

Some results of the empirical study:

Conclusion: Names as windows into society

This brief exploration of socio-onomastics, combined with modern tools like Large Language Models (LLMs), demonstrates how much information can be inferred from something as seemingly simple as a name. By leveraging contemporary AI tools, we can uncover patterns tied to cultural, historical, and demographic factors.

While the results are promising, this work also underscores the inherent challenges and limitations of using AI for such tasks. Difficulties in handling unisex names, the timeless nature of religious naming traditions, and the prevalence of shared surnames in certain cultures highlight the complexity of this field. These challenges serve as a reminder that names are deeply intertwined with societal and historical contexts, which cannot always be fully captured by even the most advanced models.

Nonetheless, the integration of LLMs into this area of research presents exciting opportunities. In contexts where breadth of analysis is prioritized over depth, LLMs provide an effective means to scale investigations across vast corpora of text. Their ability to process large datasets efficiently enables broad analyses that complement traditional approaches. This synergy could open new avenues for insights in fields such as historical research through prosopography and demographic enrichment.

Ultimately, this blog represents a small step in understanding the role of names in society. While the findings suggest that names continue to offer valuable perspectives on identity and culture, we hope this work inspires further exploration at the intersection of sociology and AI research. Interestingly, this direction was not our original focus; we initially stumbled upon this topic while seeking to enrich a dataset with additional features to solve a practical prediction task.

[paper] Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models

Sat, 02 Nov 2024 00:00:00 +0000

[paper] Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models

GitHub repo of the paper.

Summary of the paper

The paper introduces STORM, a framework designed to help large language models (LLMs) create Wikipedia-like articles from scratch by focusing on a structured pre-writing phase. STORM’s core method involves synthesizing topic outlines through a unique process of perspective-guided question asking and multi-turn simulated conversations. These steps allow the LLM to gather information from diverse viewpoints, leading to comprehensive and organized outlines. Using these outlines, STORM generates well-structured articles with improved depth, coherence, and factual accuracy. The framework outperforms traditional retrieval-augmented approaches, though challenges like source bias and verifiability remain areas for future refinement.

Q&A with ChatGPT against the paper

Q: What are they key elements in the paper’s method to generate a high quality article?

A: The key elements of the method in the paper for generating high-quality Wikipedia-like articles involve the STORM framework, which structures the pre-writing process into several steps:

Perspective-Guided Question Asking: STORM starts by identifying multiple perspectives on a topic, helping generate diverse and in-depth questions that guide research effectively.
Simulated Conversations: By simulating a conversation between a topic expert and a Wikipedia writer, the model iteratively refines its understanding of the topic, asking follow-up questions to gather comprehensive information.
Outline Creation: The gathered information is organized into an outline using the model’s internal knowledge and curated sources. This outline serves as a structured foundation for the article.
Section-by-Section Writing: Each section of the article is generated based on the outline and relevant sources, ensuring coherence, coverage, and citation accuracy.

Q: How does STORM identify the multiple perspectives on a given topic?

A: STORM identifies multiple perspectives on a given topic by first surveying related Wikipedia articles. It retrieves the tables of contents from these articles to help pinpoint diverse viewpoints relevant to the topic. These perspectives, such as those of experts or laypeople, are then used to guide the generation of varied and specific questions. This approach ensures that the model gathers a broad, nuanced understanding of the topic, allowing it to explore and integrate different facets into the final article outline.

Q: What kind of framework is used in order to simulate the conversations?

A: STORM uses the DSPy framework to simulate conversations between a “Wikipedia writer” and a “topic expert.” This framework enables multi-turn, information-seeking dialogues, allowing the model to iteratively pose questions and gather responses. Each question-answer cycle builds on previous interactions, enhancing the depth and relevance of the information collected. Trusted online sources are incorporated to ground responses in verified data, supporting a reliable foundation for the final article outline and content generation.

Q: What are the weaknesses of this paper?

A: Some weaknesses in the paper include:

Bias and Verifiability Issues: STORM may inadvertently transfer biases from its information sources, leading to articles that sometimes lack neutrality or include unverifiable connections between facts.
Citation Limitations: Though citations are used, some generated connections between information pieces are not directly verifiable, raising accuracy concerns.
Simplified Task Scope: The framework primarily generates text-based articles, overlooking multimedia or structured data (like tables) typical of high-quality Wikipedia entries, which limits its realism for broader applications.

Q: Besides the Wikipedia use case, what could be other interesting and relevant tasks that this STORM framework could tackle?

A: Beyond Wikipedia, the STORM framework could be applied to other tasks requiring comprehensive research, structure, and neutrality, such as:

Educational Content Creation: Developing curricula, study guides, or academic summaries where balanced perspectives are essential.
Technical Documentation: Producing structured and informative guides, FAQs, or manuals for complex systems in software or engineering.
Market Analysis Reports: Compiling structured business or financial reports with varied perspectives (e.g., market trends, consumer insights, regulatory impacts).
Policy Briefs: Crafting policy documents that balance diverse viewpoints, ensuring well-rounded insights for government or NGO reports.

These tasks would benefit from STORM’s capacity to integrate varied information sources, generate outlines, and maintain organized content.

Q: What kind of questions would you suggest to ask against this paper to learn as much as possible from this paper?

A: Here are some in-depth questions to guide your learning from the paper:

Framework Design: What motivated the choice of using the DSPy framework for conversation simulation, and how does it enhance the pre-writing stage? Evaluation Metrics: How do the paper’s automatic and human evaluation metrics assess outline quality? What are their limitations? Perspective Discovery: What methods did the authors explore for identifying diverse perspectives, and could alternative approaches improve this process? Bias Handling: How does the framework address potential biases in information retrieval, and what solutions are suggested for future work? Applications Beyond Wikipedia: What specific adaptations might be necessary to apply STORM in fields like policy analysis or technical documentation?

Q: What kind of experiments do the author perform with their ablation study?

A: In their ablation study, the authors evaluate the impact of different components within the STORM framework on the quality of generated Wikipedia-like articles. They systematically remove or modify specific elements to assess their contributions:

Perspective-Guided Question Asking: They examine the effect of generating questions from multiple perspectives versus a single perspective, analyzing how this influences the depth and breadth of the collected information.
Simulated Conversations: The study assesses the role of multi-turn conversations between simulated roles (e.g., “Wikipedia writer” and “topic expert”) in enhancing the relevance and coherence of the gathered content.
Outline Generation: They investigate the importance of structured outline creation prior to writing, comparing outcomes with and without this step to determine its impact on the organization and coverage of the final article.

Q: What’s the conclusion of this ablation study?

A: The study concludes that each component of the STORM framework plays a vital role in producing high-quality, well-structured Wikipedia-like articles. Removing any of these elements results in a noticeable decline in the overall quality of the generated content.

While exploring frameworks implementing similar ideas, I came across GPT Researcher and decided to give it a try. It’s straightforward to install locally, and you can start experimenting right away using the default configuration—assuming you have API keys for OpenAI and Tavily.

Setting up the experiment is as easy as this:

Wait for 5 minutes while the Agent is doing its thing…

The results were somewhat disappointing (using only the web; not grounded in a relevant corpus). Not too surprising given the sources the Agent selected…

I’ll let you judge for yourself:

Agent Output:

Building an Equity Market Neutral Strategy Using Alternative Data

In the evolving landscape of quantitative finance, equity market neutral (EMN) strategies have emerged as a robust approach to achieving returns independent of market direction. These strategies, primarily employed by hedge funds, involve taking offsetting long and short positions in closely related stocks to hedge against market exposure. This approach seeks to capitalize on price discrepancies while maintaining a balanced portfolio that is theoretically immune to market volatility. The advent of alternative data has further revolutionized this domain, offering new avenues for enhancing the precision and effectiveness of EMN strategies.

Equity market neutral strategies are designed to exploit differences in stock prices by maintaining an equal amount of long and short positions in stocks that share similar characteristics, such as market capitalization or sector affiliation. This strategy is particularly appealing to institutional investors seeking to outperform traditional bonds without the high risk associated with more aggressive funds. For a comprehensive understanding of EMN strategies, Investopedia provides a detailed overview of their mechanics and benefits.

The integration of alternative data into EMN strategies represents a significant advancement in portfolio management. Alternative data, which includes non-traditional sources such as social media sentiment, satellite imagery, and transaction data, offers a granular view of market dynamics that traditional data sources might overlook. This data can enhance the predictive accuracy of stock price movements, thereby refining the selection of long and short positions within an EMN framework. Studies, such as those published in MDPI, have demonstrated the efficacy of using machine learning models like LSTM for stock prediction, leveraging alternative data to construct market-neutral portfolios.

Moreover, the systematic application of alternative data in EMN strategies can help mitigate risks and enhance returns. By employing sophisticated algorithms and machine learning techniques, portfolio managers can dynamically adjust their positions to maintain market neutrality, even as market conditions fluctuate. This approach is supported by insights from BlackRock, which emphasizes the importance of a systematic process in capturing alpha opportunities and managing risk exposures.

In conclusion, building an equity market neutral strategy using alternative data involves a nuanced understanding of both traditional financial principles and cutting-edge data analytics. As the financial markets continue to evolve, the integration of alternative data into EMN strategies will likely become an essential component for investors seeking to achieve consistent returns in an increasingly complex market environment.

Understanding Equity Market Neutral Strategy

Equity Market Neutral (EMN) strategies are designed to exploit pricing inefficiencies while maintaining a neutral exposure to the overall market. This report explores how to build an EMN strategy using alternative data, focusing on the integration of machine learning models, data collection techniques, and risk management practices.

Leveraging Alternative Data in EMN Strategies

Alternative data refers to non-traditional data sources that can provide unique insights into market trends. These include social media sentiment, satellite imagery, and web traffic data. The use of alternative data in EMN strategies can enhance the predictive power of models and uncover alpha opportunities that traditional data might miss.

Data Collection and Processing

The first step in incorporating alternative data into an EMN strategy is data collection and processing. Hedge funds and investment firms gather a wide array of data, including financial statements, earnings reports, and alternative unstructured data like news articles and social media sentiment (Aurum). The data is then cleaned and structured to be fed into quantitative models.

Signal Generation

Once the data is processed, the next step is generating trading signals. Quantitative models analyze the data to identify patterns and predict future stock movements. For instance, a model might use natural language processing (NLP) to gauge sentiment from news articles, which can then be used to predict stock price movements (Aurum). This approach allows for the creation of signals based on value, growth, price momentum, and quality.

Machine Learning Models in EMN Strategies

Machine learning models, particularly Long Short-Term Memory (LSTM) networks, have gained popularity in EMN strategies due to their ability to handle time series data effectively. These models can predict stock prices and returns with high accuracy, which is crucial for constructing a market-neutral portfolio (MDPI).

LSTM-Based Stock Prediction

LSTM networks are particularly suited for stock prediction due to their ability to capture long-term dependencies in data. By training on historical stock prices and alternative data, LSTM models can forecast future price movements, aiding in the selection of stocks for long and short positions in an EMN strategy.

Portfolio Construction

The construction of a market-neutral portfolio involves balancing long and short positions to achieve a beta close to zero. This is typically done by selecting an equal number of stocks for both positions, ensuring that gains from one side offset losses from the other (MDPI). The use of LSTM models in this process can enhance the robustness of the portfolio by improving stock selection accuracy.

Risk Management in EMN Strategies

Risk management is a critical component of EMN strategies, as it ensures that the portfolio remains market-neutral and minimizes exposure to systematic risk. This involves continuous monitoring and rebalancing of positions to maintain the desired risk profile.

Hedging Techniques

One of the main advantages of EMN strategies is their ability to hedge against market risk. By taking long positions in stocks expected to outperform and short positions in those expected to underperform, the strategy can mitigate the impact of market volatility (Picton Mahoney).

Diversification Benefits

EMN strategies offer diversification benefits by providing returns that are independent of market direction. This can be particularly valuable in periods of high market volatility, where traditional portfolios may suffer significant losses. By incorporating alternative data and machine learning models, EMN strategies can further enhance diversification by uncovering unique alpha opportunities (BlackRock).

Performance Evaluation and Adjustment

Evaluating the performance of an EMN strategy involves analyzing its risk-adjusted returns and ensuring that it meets the desired investment objectives. This requires regular backtesting and adjustment of models and data inputs to adapt to changing market conditions.

Backtesting and Model Validation

Backtesting is a crucial step in validating the effectiveness of an EMN strategy. By simulating the strategy’s performance on historical data, investors can assess its potential returns and identify any weaknesses in the model. This process also involves adjusting the model parameters and data inputs to optimize performance (CAIA).

Continuous Improvement

The dynamic nature of financial markets necessitates continuous improvement of EMN strategies. This involves updating models with new data, incorporating additional alternative data sources, and refining trading algorithms to maintain a competitive edge. By staying abreast of technological advancements and market trends, investors can ensure that their EMN strategies remain effective and profitable (Class Central).

Integration with Traditional Investment Strategies

While EMN strategies offer unique benefits, they can also complement traditional investment strategies by providing additional diversification and risk mitigation.

Complementing the 60/40 Portfolio

The traditional 60/40 portfolio, which allocates 60% to equities and 40% to bonds, can benefit from the inclusion of EMN strategies. By providing returns that are uncorrelated with the broader market, EMN strategies can enhance the overall risk-adjusted returns of the portfolio and reduce its vulnerability to market downturns (CAIA).

Enhancing Long-Biased Portfolios

EMN strategies can also complement long-biased portfolios by offering a hedge against market declines. By balancing long and short positions, these strategies can mitigate the impact of market volatility and provide a more stable return profile (Picton Mahoney).

In summary, building an equity market-neutral strategy using alternative data involves integrating innovative data sources, advanced machine learning models, and robust risk management practices. By doing so, investors can create strategies that offer compelling returns and diversification benefits, even in challenging market conditions.

Utilizing Alternative Data in EMN Strategy Advanced Data Integration Techniques

While previous sections have covered the basics of data collection and processing, this section delves deeper into advanced techniques for integrating alternative data into Equity Market Neutral (EMN) strategies. Unlike traditional data sources, alternative data such as satellite imagery, social media sentiment, and web traffic data require sophisticated integration methods to be effectively utilized in quantitative models.

Data Fusion: This technique involves combining multiple data sources to create a comprehensive dataset that enhances the predictive power of EMN strategies. For example, integrating satellite imagery with social media sentiment can provide a more nuanced view of consumer behavior and market trends (AlphaSense). Natural Language Processing (NLP): NLP is used to analyze unstructured text data from news articles and social media posts. By extracting sentiment and key themes, NLP can generate valuable insights that inform trading decisions. This approach is particularly useful for capturing market sentiment shifts that may not be reflected in traditional financial metrics (Aurum). Geospatial Analysis: Utilizing geolocation data from smartphones and other devices can offer insights into foot traffic patterns and consumer behavior. This data can be particularly valuable for retail and consumer goods sectors, providing a real-time view of market dynamics (AlphaSense). Signal Enhancement through Machine Learning

While previous sections discussed LSTM-based stock prediction, this section focuses on enhancing signal generation through advanced machine learning techniques. These methods leverage alternative data to improve the accuracy and reliability of trading signals in EMN strategies.

Ensemble Learning: By combining predictions from multiple machine learning models, ensemble learning can enhance the robustness of trading signals. This approach mitigates the risk of overfitting and improves the generalization of models to new data (OptimizDBA). Feature Engineering: Advanced feature engineering techniques can extract meaningful patterns from alternative data, such as identifying correlations between social media sentiment and stock price movements. These engineered features can significantly enhance the predictive power of machine learning models (SSRN). Deep Reinforcement Learning: This technique involves training models to make sequential decisions by interacting with a simulated market environment. By learning from past actions and outcomes, reinforcement learning models can optimize trading strategies in dynamic market conditions (Academia.edu). Risk Management Innovations

While previous sections have covered traditional hedging techniques, this section explores innovative risk management strategies that leverage alternative data to enhance EMN strategies.

Dynamic Risk Assessment: By continuously monitoring alternative data sources, such as news sentiment and social media trends, investors can dynamically assess and adjust risk exposures. This proactive approach allows for timely responses to emerging risks and opportunities (BlackRock). Volatility Prediction: Using machine learning models to predict market volatility based on alternative data can help investors better manage risk. For instance, analyzing social media sentiment can provide early warnings of potential market disruptions, allowing for preemptive risk mitigation (MDPI). Scenario Analysis: By simulating various market scenarios using alternative data, investors can evaluate the potential impact of different risk factors on their portfolios. This approach enables more informed decision-making and enhances the resilience of EMN strategies (OptimizDBA). Portfolio Optimization with Alternative Data

While previous sections have discussed portfolio construction, this section focuses on optimizing EMN portfolios using alternative data to achieve superior risk-adjusted returns.

Multi-Objective Optimization: This technique involves optimizing portfolios based on multiple criteria, such as maximizing returns while minimizing risk. By incorporating alternative data, investors can identify non-traditional factors that contribute to portfolio performance, such as consumer sentiment and environmental trends (Aurum). Adaptive Rebalancing: Utilizing real-time alternative data allows for adaptive portfolio rebalancing, ensuring that portfolios remain aligned with current market conditions. This approach enhances the agility of EMN strategies and improves their ability to capture short-term market opportunities (Academia.edu). Factor Analysis: By analyzing alternative data, investors can identify new factors that drive stock returns, such as brand loyalty or supply chain disruptions. Incorporating these factors into portfolio optimization models can enhance the predictive accuracy and performance of EMN strategies (SSRN). Continuous Strategy Refinement

While previous sections have emphasized the importance of continuous improvement, this section highlights specific strategies for refining EMN strategies using alternative data.

Feedback Loops: Establishing feedback loops that incorporate alternative data allows for the continuous refinement of trading models. By analyzing the outcomes of past trades, investors can identify areas for improvement and adjust their strategies accordingly (OptimizDBA). Data-Driven Insights: Leveraging alternative data for insights into market trends and consumer behavior can inform strategic adjustments to EMN strategies. This data-driven approach ensures that strategies remain relevant and effective in changing market environments (AlphaSense). Collaborative Innovation: Collaborating with data scientists and industry experts can foster innovation and enhance the development of EMN strategies. By sharing insights and expertise, investors can leverage the full potential of alternative data to drive superior performance (Aurum). Portfolio Construction and Risk Management in Equity Market Neutral Strategies Factor-Based Stock Selection

In equity market neutral (EMN) strategies, selecting the right stocks is crucial for achieving market neutrality and optimizing returns. Unlike traditional methods that rely heavily on historical price data, EMN strategies can benefit significantly from alternative data sources. These data sources include social media sentiment, satellite imagery, and web traffic analytics, which provide insights into market trends and company performance that are not captured by conventional financial metrics. For instance, social media sentiment analysis can reveal public perception and potential market movements before they are reflected in stock prices (CAIA).

Advanced Machine Learning Techniques

While LSTM models have been discussed in previous sections for stock prediction, this section focuses on other machine learning techniques that can enhance EMN strategies. Techniques such as random forests, gradient boosting machines, and support vector machines can be employed to capture complex, non-linear relationships in the data. These models can process large volumes of alternative data to identify patterns and predict stock movements more accurately. For example, random forests can be used to analyze a combination of financial ratios and alternative data to rank stocks based on their expected returns, thereby aiding in the construction of the long and short sides of the portfolio (Academia.edu).

Dynamic Portfolio Rebalancing

Dynamic portfolio rebalancing is a key component of risk management in EMN strategies. Unlike static rebalancing, which occurs at fixed intervals, dynamic rebalancing adjusts the portfolio in response to real-time market changes and new data inputs. This approach ensures that the portfolio remains aligned with the strategy’s objectives and market conditions. By integrating alternative data, such as real-time economic indicators and sentiment analysis, portfolio managers can make more informed decisions about when to rebalance the portfolio. This not only helps in maintaining market neutrality but also in capturing short-term market opportunities that may arise due to sudden shifts in market sentiment (MDPI).

Risk Metrics Beyond Traditional Measures

Traditional risk metrics like standard deviation and beta may not fully capture the risks associated with EMN strategies. Instead, advanced risk metrics such as Conditional Value-at-Risk (CVaR) and entropy pooling offer a more comprehensive view of potential risks. CVaR, for instance, measures the expected loss in the worst-case scenarios, providing a more robust assessment of tail risks. Entropy pooling, on the other hand, allows for the incorporation of subjective views and stress-testing scenarios, enabling a more nuanced risk management approach. These metrics help in identifying and mitigating risks that are not apparent through traditional measures, thereby enhancing the robustness of EMN strategies (Anton Vorobets).

Incorporating Sentiment Analysis

Sentiment analysis is an innovative tool for enhancing EMN strategies by providing insights into market psychology and investor behavior. By analyzing data from news articles, social media platforms, and financial reports, sentiment analysis can gauge the overall market mood and predict potential market movements. This information can be used to adjust the portfolio’s long and short positions to better align with market sentiment. For instance, a positive sentiment towards a particular sector may prompt an increase in long positions within that sector, while negative sentiment may lead to an increase in short positions. This proactive approach to portfolio management can improve the strategy’s performance and resilience in volatile markets (CAIA).

Integration of ESG Factors

Environmental, Social, and Governance (ESG) factors are increasingly being integrated into EMN strategies as a form of alternative data. ESG factors provide insights into a company’s sustainability practices and ethical considerations, which can influence long-term performance. By incorporating ESG data, EMN strategies can identify companies that are likely to outperform due to their strong ESG profiles. This not only enhances the strategy’s return potential but also aligns it with socially responsible investment practices. For example, companies with high ESG ratings may be favored for long positions, while those with poor ratings may be targeted for short positions, thereby aligning financial goals with ethical considerations (MDPI).

Continuous Monitoring and Adjustment

Continuous monitoring and adjustment are critical for maintaining the effectiveness of EMN strategies. This involves regularly reviewing the portfolio’s performance, risk exposures, and alignment with market conditions. By leveraging alternative data and advanced analytics, portfolio managers can identify emerging trends and potential risks in real-time. This enables them to make timely adjustments to the portfolio, such as reallocating assets or adjusting hedging strategies, to ensure that it remains on track to achieve its objectives. Continuous monitoring also facilitates the identification of underperforming assets, allowing for their timely replacement with better-performing alternatives (Academia.edu).

By integrating these advanced techniques and data sources, equity market neutral strategies can be significantly enhanced, providing better risk-adjusted returns and resilience in diverse market conditions. This approach not only leverages the latest advancements in data analytics and machine learning but also aligns with the growing demand for sustainable and responsible investment practices.

Conclusion

This research report outlines the construction of an Equity Market Neutral (EMN) strategy utilizing alternative data, emphasizing the integration of advanced machine learning models, sophisticated data collection techniques, and innovative risk management practices. The primary aim of EMN strategies is to exploit pricing inefficiencies while maintaining a neutral exposure to market movements. By leveraging alternative data sources such as social media sentiment, satellite imagery, and web traffic analytics, these strategies can enhance predictive accuracy and uncover alpha opportunities that traditional data might overlook. The report highlights the importance of data processing and signal generation, where techniques like Natural Language Processing (NLP) and geospatial analysis play crucial roles in extracting actionable insights from unstructured data.

The incorporation of machine learning models, particularly Long Short-Term Memory (LSTM) networks, is pivotal in handling time series data and predicting stock movements with high precision. These models, along with other techniques like ensemble learning and deep reinforcement learning, significantly improve the robustness and reliability of trading signals. Risk management is another critical component, with dynamic risk assessment and volatility prediction using alternative data providing a proactive approach to mitigating market risks. The report also underscores the benefits of integrating Environmental, Social, and Governance (ESG) factors, which align financial objectives with ethical investment practices, thereby enhancing the strategy’s appeal to socially responsible investors.

The findings suggest that the integration of alternative data and advanced analytics can significantly enhance the performance and resilience of EMN strategies, offering superior risk-adjusted returns even in volatile market conditions. The next steps involve continuous refinement of these strategies through feedback loops, data-driven insights, and collaborative innovation with data scientists and industry experts. By staying abreast of technological advancements and market trends, investors can ensure that their EMN strategies remain competitive and effective. This approach not only leverages cutting-edge data analytics but also aligns with the growing demand for sustainable and responsible investment practices, positioning EMN strategies as a valuable complement to traditional investment portfolios.

Conclusion: Fully automated generation (at least, this framework) does not seem yet on par with co-creation with a human expert, but a decent entrypoint for the layman.