FinEval 2026: The Research Behind Our Competition Results

I am writing this before flying to Budapest for EMNLP 2026, where I will also attend the FinNLP workshop. Khaled Al Nuaimi and I entered the three FinEval tasks under the team name ADML. Before the conference, I wanted to write up what we tried, including the experiments that did not work.

The results were good: first in Greek financial named entity recognition, tied first in Japanese implicit communication recognition, and second in multilingual financial question answering. We ended up submitting historical answer lookups for two tasks and Japanese response rules for the third. After all the model training, none of our final submissions needed a neural model at inference time.

I find that outcome somewhat disappointing. We had investigated much more interesting questions: how to correct entity boundaries without introducing new errors; how to recognize commitment in an indirect answer; how to combine financial statements and multilingual news into a short, verifiable answer. Some approaches improved our scores. Others looked promising on one validation set and failed on the next.

This post covers those experiments and why we eventually submitted much simpler systems.

The code, three system papers, research logs, and conference materials are public. The rankings on the official FinEval website list our entries under my account, gmarti.

Three tasks and three initial hypotheses
Greek NER: from recognizing numbers to learning annotation decisions
Japanese intent: the difficulty was in the boundaries
Financial QA: correct arithmetic was only the beginning
Where we stood before the test release
The release and feedback that changed our submissions
The private results complicated the ending
The research I want to continue

Three tasks and three initial hypotheses

The FinEval shared task covered three quite different problems.

Task	What the system had to do	Evaluation
Greek financial NER	Extract people, organizations and locations, or financial numerical expressions, from Greek reports	Entity precision, recall and F1
Japanese Financial Implicit Communication Recognition, or JF-ICR	Read a question and management’s answer, then classify the strength and direction of commitment	Five-class accuracy
PolyFiQA	Answer financial questions using statements and news supplied in English, Chinese, Japanese, Spanish and Greek	ROUGE-1 F1, with a 100-word answer limit

The Greek task has two sets of entity labels. GRFinNER covers textual entities; GRFinNUM covers money, percentages, temporal expressions, quantities and other numerals. A sentence can occur under either instruction, so deciding which extraction task is being requested is part of the pipeline.

The Japanese labels run from strong refusal, −2, through qualified refusal, −1, and neutral or unresolved intent, 0, to qualified commitment, +1, and strong commitment, +2. This is more subtle than sentiment. A manager can describe a difficult market while firmly committing to an investment, or express enthusiasm without committing to anything.

PolyFiQA combines financial tables with multilingual news. Its Easy questions concern revenue trends, balance-sheet health, cash flows and R&D ratios. Its Expert questions concern revenue priorities, capital allocation, margin strategy and capital expenditure. Even with the documents already supplied, the system has to select the relevant facts, identify the right periods and units, calculate where necessary, and decide what deserves space in a short answer.

Our initial research directions followed these differences. For Greek, we could start with language-specific encoders and improve exact extraction. For Japanese, we needed to understand the relationship between a question, an event and the speaker’s commitment to it. For QA, we wanted answers grounded in the supplied financial statements, with numbers and calculations we could verify against the source documents.

First, we had to establish what data we had. The Japanese description referred to a 94-example core, while the downloaded release contained 253 labeled examples. The initially available QA file contained 76 Easy questions, arranged as 19 report bundles with four questions each. We recovered additional textual Greek and Expert QA resources from the public benchmark repositories. Looking at those releases also exposed annotation anomalies and the possibility of historical overlap, well before the current competition test appeared.

Greek NER: from recognizing numbers to learning annotation decisions

The numeric track gave us the first clear wins

We began with a CPU rule baseline. Financial reports contain many recognizable patterns: percentages, currency amounts, dates, registry identifiers and quantities. But recognizing a number is easier than predicting the exact surface and category expected by an annotation scheme. Should a percentage include its symbol? Is a date one entity or several? Is a bare number a quantity or an identifier?

Aligning the rules with these conventions moved our local validation F1 from about 0.56 to 0.7491. Fine-tuning Greek BERT then produced a much larger gain.

An interesting find was GreekLegalBERT v2. Its pretraining corpus includes Greek legislation, EU legal documents and parliamentary proceedings. Corporate filings contain plenty of legal and regulatory language, formal company names and registration details, so a legal encoder seemed worth trying for financial extraction. Fine-tuning it on our numeric task reached 0.9619, compared with 0.9410 for the general Greek BERT in the seed-7 runs: about 2.1 F1 points higher.

We had first found an NER-fine-tuned version, replaced its classification head to match our labels, and obtained 0.9610. Going back to the underlying legal encoder did slightly better. I liked this connection between legal NLP and corporate finance: the language of filings gave us a reason to try a model we might have missed by searching only for financial models.

Here are the numeric extraction results:

Numeric system	Local validation entity F1
Rules after annotation alignment	0.7491
Greek BERT, seed 7	0.9410
GreekLegalBERT v2, seed 7	0.9619
Two-model legal ensemble	0.9636
Legal ensemble with a finance-adapted view	0.9645
Identifier recovery	0.9832
Identifier and calendar recovery	0.9850

These are development scores on the 80-row numeric validation set, averaging entity F1 over rows. The official leaderboard pools entities, so its scores later in this post are a different aggregation. The successive development experiments all reused this validation set.

The last gains came from inspecting concrete omissions. The ensemble missed Greek registry identifiers after recognizable auditor, tax and company-register prefixes. We checked the patterns against real Athens Stock Exchange filings: the final identifier expression matched 850 occurrences across 241 documents. A calendar rule recovered a day number immediately before a Greek month name, supported by all 58 matching training occurrences and all 19 validation occurrences.

We also found a rule that could repair almost the last validation miss, involving a zone designator. It remained a secondary experiment because the exact rule had been motivated by that miss and lacked direct labeled training support.

Domain adaptation was more ambiguous. Continuing language-model pretraining on financial filings did not produce the strongest individual numeric model, but it supplied useful ensemble diversity. Expanding and cleaning that corpus further did not yield another ensemble gain.

Textual entities exposed the annotation problem

The textual track started around 0.8573 validation F1 with a Greek legal BERT. Combining legal and finance-adapted models raised the flat ensemble to 0.8677. Many remaining errors were recognizable entities with the wrong boundary, nesting or output spelling.

There was also a bug in the released answers. Uppercase Latin PER inside an organization name had been replaced by the Greek word for the PERSON label, ΠΡΟΣΩΠΟ. Thus PAPERLESS could appear in an answer as PAΠΡΟΣΩΠΟLESS. A model extracting the actual company name from the sentence could be semantically correct and still lose under exact matching.

We first repaired alignment so the model could train on these examples. Then we added a narrow output conversion reproducing the observed release convention. That alone moved the flat ensemble from 0.8677 to 0.9025 on validation, entirely through the output spelling conversion.

We also tried extracting nested entities. A location can occur inside an organization name (think of “Greece” in “Bank of Greece”). Our tagger assigned one label per token, so it could not mark both. We tried separate prediction heads for the two labels, restricting the extra locations to spans inside predicted organization names.

Initially, this looked promising. Before the output-encoding correction, the nested addition raised validation F1 from 0.8677 to 0.8860. But when the frozen systems were evaluated on the previously held-out historical public test, the ordering reversed: the flat system scored 0.8432, while the nested version scored 0.8314.

The validation gain did not carry over to the held-out test, so we did not keep the broader nested-entity expansion. We retained only narrower nesting rules supported by the training annotations.

Learning which corrections to accept

The next direction was to treat a second model as a proposer of corrections. GLiNER supplied candidate spans, while the existing taggers supplied a strong baseline. We trained a model to accept or reject individual edits: adding a span, changing its boundary, or correcting its type.

We kept related occurrences of person and organization names in the same fold, so a familiar name could not appear on both sides of the training/evaluation split. Each row was predicted by models that had not trained on it. We then trained a correction model on their disagreements. It improved the out-of-fold baseline from 0.8197 to 0.8594. It helped 38 rows and hurt 11, so there was useful signal but still a cost to intervention.

For our consensus system, we required several models to agree on a correction. Models trained on the full data, individual folds and a second random seed sometimes proposed different boundaries. Requiring agreement filtered out changes proposed by only one model. Combining accepted operations with structural repairs raised the textual validation baseline from 0.9208 to 0.9332, improving four rows and harming none.

I found this result interesting: we could improve a strong tagger by learning which individual corrections to accept. Simply merging all the predicted entities was less useful.

Several attempts to extend it failed. A neural boundary ranker did not pass its first-fold test. Other correction models learned useful scores in calibration but could not transfer their thresholds reliably to deployment. An entity-aware pretraining experiment, which masked whole entities in external financial text, fell from 0.8064 to 0.7580 against the unchanged encoder on its first held-out fold. All three entity types regressed, and the experiment stopped there.

The Greek research log records the intermediate models and the rejected branches.

Japanese intent: the difficulty was in the boundaries

A deceptively strong majority baseline

In the Japanese task, always predicting qualified commitment, +1, already gave a fairly high accuracy:

Label	Meaning	Released examples
+2	Strong commitment	29
+1	Qualified commitment	139
0	Neutral or unresolved	72
−1	Qualified refusal	11
−2	Strong refusal	2

The majority accuracy was 139/253, or 0.5494. On the original 94-example core, the same majority occupied about 69% of the data. A model could look respectable while learning very little about refusal.

Our first lexical classifiers and multilingual embedding classifiers did not solve this. Their apparent usefulness also depended on how we split the data. We reconstructed source groups from company documents, consolidating examples that belonged to the same source even when their IDs were far apart. Some source assignments were inferred from document continuity, so this was an imperfect but more demanding evaluation than random rows.

Under reconstructed source holdout, the early TF-IDF classifier fell to 0.5336, below the majority baseline. We moved on to models trained on question–response pairs.

We also tried direct LLM judgment, structured decomposition and contrastive examples. Recovering the original annotation guidelines improved a diagnostic pilot, especially by clarifying that background explanation alone should not be treated as commitment. But the pilot was deliberately enriched for rare negative labels, and broader results did not establish a reliable advantage.

A small Japanese encoder and a useful ordinal change

Fine-tuning Tohoku Japanese BERT on question–response pairs gave us a stronger base. Two fixed seeds, averaged together, reached 0.6561 under source-disjoint validation. Adding previously frozen conservative rules raised that to 0.6719.

Then a simple decoding change helped. The five labels are ordered. Instead of choosing the single most probable class, we calculated the expected label on the scale −2 to +2 and rounded it. For example, probability spread between neutral and strong commitment can support an intermediate commitment level. This changed no model weights and fitted no new threshold.

Accuracy rose to 0.6917, with seven corrected predictions and two newly incorrect ones. Four source folds improved and the fifth tied. That became our main learned system.

We tried several ways to improve it: train on the ordering of the labels, adapt the encoder to financial text, add examples of promises, and use annotators’ highlighted phrases as extra supervision.

Finance adaptation, promises and human rationales

For domain adaptation, we trained on 50,000 masked-language-model blocks from an EDINET financial corpus, excluding benchmark-company matches. The language model’s held-out loss improved substantially. The downstream result was less conclusive: one blend reached 0.7036, above 0.6917, but its gains were uneven across sources and did not meet our criteria for replacing the model. A procedure that selected the blend weight without using the held-out source actually fell to 0.6838.

We then explored external promise supervision. After excluding an overlapping issuer and near-duplicates, a Japanese promise dataset supplied 357 auxiliary paragraphs. It could help locate statements about future action. But using only the selected event discarded useful surrounding context, and adding its features to the incumbent did not improve accuracy. A multitask model trained to classify intent, order commitments, detect promises and locate their spans also failed its first-fold comparison.

The annotators’ explanations looked promising. An original annotation export contained exact response spans associated with the annotators’ decisions. We used a training-only token head to teach the encoder where to look, alongside its ordinary intent objective.

On the first fold, a three-model blend improved from 34/56 to 38/56, correcting five errors while creating one. We then tested that same approach on another source fold. It went from 30/51 to 27/51, correcting none and creating three errors. The promising result did not replicate.

Why diverse models were not enough

We tried alternate Japanese encoders, NLI intermediate training, ordinal heads, distance-aware losses and selectors that chose among frozen experts. Different models got different examples right. We wanted to learn which model to trust for a given question and response.

For example, a collection of five encoders had an accuracy oracle of 0.7787: that is the score we could obtain if we already knew which model was right for each example. A selector based on local lexical expertise reached only 0.6838, below our 0.6917 baseline.

A distance-aware loss produced another near-success. A four-model diagnostic ensemble reached 181/253, or 0.7154, with six corrections and no new errors relative to the incumbent. But the experiment also required an independent-seed replication. That replication tied the incumbent, with five corrections and five new errors. We retained the original system.

We also tried adding refusal rules. We collected unrelated-company IR Q&A and derived refusal expressions outside the benchmark. Applying them as overrides corrected two validation rows and damaged fifteen, reducing accuracy to 0.6403. All five source folds worsened.

Refusing to disclose details is different from refusing to undertake the action in the analyst’s question. Negating a subordinate clause is different from rejecting the main proposition. Words such as “no plan” or “cannot disclose” do not determine the overall answer without scope and context.

What I would still like to try is to identify the action in the question, then determine whether the response commits to it, under which conditions, and on what timescale. We had tested parts of this approach without getting a reliable improvement. The Japanese experiments contain considerably more failed controls than the final two-seed model might suggest.

Financial QA: correct arithmetic was only the beginning

Building an answer from facts we could verify

For PolyFiQA, we began by exploiting the repeated question families. A report bundle contains the same statements and news for four questions. That makes it possible to build specialized extractors and calculators before asking a language model to generate an answer.

We parsed statement rows into facts carrying a metric, value, unit, reporting period and source span. Derived values retained their operands and operation. For an R&D ratio, for example, the system needed R&D expenditure and revenue for the same period, in compatible units. Accounting parentheses had to become negative numbers; a missing total could sometimes be reconstructed from its components or the balance-sheet identity.

The resulting pipeline was a small financial compiler: parse the input, identify the question family, select facts, perform permitted arithmetic, and render a concise answer with evidence. Every emitted number had to come from the supplied material or a supported calculation.

When an answer was wrong, we could trace it back through the calculation: had we selected the wrong period, confused a stock with a flow, or misread a unit? We could also check which news passages had been included.

It also exposed problems in the references. Some released answers appeared cyclically assigned to the wrong questions: a balance-sheet question could receive an R&D answer, for example. We kept the raw releases intact and reported diagnostics on identifiable aligned subsets separately. Those subsets helped investigate the system, but were not substitutes for an official test score.

Retrieval needed financial constraints

Retrieving semantically related news was useful for some questions and harmful for others. Revenue questions often benefited from news evidence. Cash-flow answers could become worse when we appended an approximately relevant sentence that did not explain the statement’s operating, investing or financing flows.

The useful improvement was often to make retrieval more specific. For revenue, we compared a candidate news sentence’s amount with total statement revenue, checked its year-over-year percentage, and demoted segment-level statements when the question concerned the whole company. The same amount could appear with English units, Spanish or Greek separators, Chinese units, or a compound Japanese expression. Normalizing those forms let numerical agreement guide the language retrieval.

For R&D, we required an explicit research-and-development expression and a financial cue. Generic language about innovation did not qualify. We also derived reporting periods from statement structure, abstaining when a fiscal calendar did not support the inference.

The wording mattered too. Spelling out “million” and “billion” helped cash-flow answers. Applying the same change everywhere could hurt: longer revenue wording displaced more useful evidence under the word limit. A locally sensible edit had to be evaluated in the complete answer.

One important correction concerned the evaluator itself. Earlier diagnostics used a generic ROUGE implementation. Recovering the organizer-compatible multilingual tokenizer changed the interpretation of several results, so we rescored stored candidates. In particular, a long Japanese passage can count as one whitespace-delimited word for the submission limit while contributing many character tokens to ROUGE. We eventually limited the second CJK revenue passage using the scorer’s token count as well as the submission word count.

On the same 192-row historical evaluation, the incremental progress was:

Input-grounded QA system	Historical ROUGE-1 F1
Earlier complete baseline	0.3266
Explicit cash-flow units	0.3281
Selective multilingual R&D evidence	0.3375
Statement-derived R&D periods	0.3409
Revenue retrieval grounded in statement amounts	0.3493
Later balance, cash-flow, margin and evidence-budget refinements	0.3652

We generated these answers from the supplied documents and scored them against historical references. All versions used the same 192 examples. The last change improved the overall score but reduced it for one company.

Generating a better candidate did not solve selection

We did try language models. Direct generation and conservative editing with Eva and Qwen variants sometimes produced a better answer, but the paired comparisons did not support wholesale replacement. In one 32-row pilot, a Qwen 35B editor scored 0.2829, against 0.3465 for the deterministic answers it was supposed to improve.

We then gave the models a graph of verified financial facts and asked them to select fact IDs for the answer. Larger 35B and 57B planners recovered more relevant numbers on questions whose references actually contained numeric targets. Across the full screen, however, they selected too many facts for qualitative answers and lost on macro fact-set F1.

A follow-up precision filter improved aggregate fact-set F1 on a separate 28-row confirmation set, from 0.7454 to 0.7638, but one held-out issuer lost 0.0544. We had required the filter to pass a check for each issuer, so we did not keep it. These scores measure fact selection, not ROUGE on the generated answers.

We also tried a smaller action space: keep the baseline answer, remove a sentence, retain only its core, retain only its evidence, or substitute a fully validated alternative. These sparse edits gave the selector fewer ways to introduce a mistake.

If we used the reference to choose the best candidate for every row, the score increased from 0.3652 to 0.4030, improving 139 of 192 answers. But that is an oracle: it already knows the answer we are evaluated against.

The deployable selector, evaluated with companies held out, chose to leave every answer unchanged. A separate temporal-transfer experiment trained on older packets from other issuers changed four newer answers and harmed all four. Having a better candidate somewhere in a small list was much easier than identifying it from the question, evidence and candidate text.

A revenue compiler improved a core-only proxy from 0.3297 to 0.3350. When we evaluated the complete answers, including their evidence, it reduced ROUGE from 0.3586 to 0.2563. We dropped that change too.

By then, the QA research had shifted from “can we calculate the right numbers?” toward “can we choose the right facts and evidence, and know when a revision will help?” Our compiler was useful within the covered schemas, but neither a large generator nor a learned edit selector had resolved that second question. The QA research log preserves these changes of direction.

Where we stood before the test release

Before the test inputs arrived, our development results looked like this:

Task	What we had established	What remained uncertain
Greek NER	Strong numeric extraction; useful textual correction through conservative consensus	Transfer of annotation conventions; repeatedly inspected validation; remaining boundary errors
Japanese ICR	A 0.6917 source-disjoint learned system and a useful ordinal decoder	Very weak support for rare refusals; unstable improvements across sources
Financial QA	A numerically checked compiler and incremental historical ROUGE gains	Reference scope and wording; selection across issuers and periods; performance on unfamiliar question schemas

Greek extraction gave us the strongest grounds for confidence. Japanese intent remained the most statistically fragile: there were only two strong refusals in the released training data. For QA, we were confident about many mechanical properties of the answers, but much less confident that a correct, grounded answer would match the reference’s choice of explanation.

We had reused small validation sets throughout, selected some leads after inspecting errors, and stopped other experiments after one fold. Several gains disappeared when we tested on another source or repeated the experiment with a different seed.

We had also already identified historical public answer resources and built separate retrieval candidates. Their existence did not tell us whether the new test would overlap, whether its annotations would remain the same, or whether the organizers would permit their use. We therefore continued developing systems that could answer from the supplied inputs.

The release and feedback that changed our submissions

We had local labeled development data throughout the research. The participant test inputs arrived on August 11. The leaderboards and their public subset scores became available on August 17. The current hidden reference answers were not released to us.

August 11: the overlap became concrete

The input audit found:

Greek NER: all 200 queries, covering 100 textual and 100 numeric examples, exactly matched historical public inputs.
Japanese ICR: 50 new question–response pairs, without exact or normalized overlap with the released training examples.
PolyFiQA: all 76 query packets exactly matched historical public inputs.

For Greek and QA, this confirmed a possibility we had anticipated. We could retrieve an old reference answer for every competition query.

The matching requirement was strict. We compared the full query, including the extraction instruction for Greek and the evidence packet for QA. An ID, company name or similar-looking question was not sufficient. Matching the same company in a different reporting period would be an obvious way to retrieve a plausible but wrong answer.

These were historical-answer lookups. They were different from the earlier QA retrieval of relevant news inside the supplied evidence. The selected memory system would copy an existing answer, with the QA word cap applied, rather than reconstruct an answer from retrieved facts.

At this point, the lookup candidates remained separate while permission was unresolved. The August 11 portfolio still selected the Greek consensus models, the Japanese ordinal ensemble and the grounded QA compiler.

August 17: the competition decision changed

The organizers then explicitly permitted additional external data, alongside training, fine-tuning, prompt rewriting and augmentation. This made the historical-reference candidates eligible under the clarified rules.

Public feedback strongly favored them. Greek retrieval scored 1.0000 entity F1, against 0.9665 for the input-based ensemble. QA retrieval scored 0.9994 ROUGE-1 F1, against 0.3214 for the input-grounded compiler.

With those public scores, we chose historical retrieval for both tasks. I would have preferred to submit the models we had developed, but expected them to rank lower. We disclosed the reuse of historical answers.

Japanese intent required a different pivot

The Japanese result was the real shock: our 0.6917 source-disjoint model scored only 4/15, or 0.2667, on the public test subset.

Inspecting the full label-free input packet revealed a substantial change in the task distribution. The source examples were skewed toward qualified commitment and neutral intent, often expressed indirectly. The new responses were unusually explicit and template-like. The competition description also specified a balanced test design, with ten examples for each of the five labels.

The responses could be classified with rules along these lines:

Intent	Example wording in English
Strong commitment	The board has approved the action; the agreement is signed
Qualified commitment	We intend to proceed if the stated conditions are met
Neutral	We are still evaluating the alternatives
Qualified refusal	We will not proceed at present, but may reconsider
Strong refusal	The action has been cancelled; we have no intention to proceed

The final code used 40 Japanese response patterns, checked in a fixed order, with neutral as the default. It parsed the prompt but classified the response text; the question-conditioned event model remained a research direction. There was no ID-to-label map. The frozen output contained ten examples per class; the balance check verified that property rather than moving uncertain examples between classes to fill a quota.

We submitted the completed rubric once. It scored 15/15 publicly, and we did not use that score to revise the rules.

We designed the rubric after the model’s poor public score led us to inspect the test inputs. It was specific to that packet. We had no hidden labels, row-level feedback or information about which rows belonged to the public subset.

Simple linguistic patterns were unreliable on the natural source responses; a compact rule system worked extremely well on the new, explicit packet.

The private results complicated the ending

The final results were:

Task and selected system	Public score	Private score	Private rank
Greek NER, historical retrieval	1.0000	0.9676	1st
Japanese ICR, explicit rubric	1.0000, 15/15	0.9714, 34/35	Tied 1st
PolyFiQA, historical retrieval	0.9994	0.3726	2nd

Japanese ICR was a three-way tie at the best private accuracy, even though our entry appeared first in the displayed ranking.

For Greek, perfect historical input coverage did not produce perfect private entity matching. The old answers were evidently not an exact match for all private evaluation expectations. Annotation revisions are a plausible explanation, but we cannot identify the affected spans without the private labels.

For QA, the contrast was much larger: 0.9994 on 24 public rows, then 0.3726 on 52 private rows, despite exact historical input coverage for all 76. Reference-version changes are our leading explanation. Identical questions and documents can be paired with answers that select different evidence, give a different amount of detail or use different wording.

We do not have the private references to check this explanation. The public and private scores also cover different subsets of questions.

The public result provided almost no reason to prefer regeneration. Nor had earlier snapshots supplied a strong warning: all 76 overlapping Expert answers were byte-identical across the two historical releases we compared. The near-perfect public score reinforced the choice to preserve that wording. It could not tell us whether the private answers followed another version.

We also cannot claim that our grounded compiler would have beaten retrieval privately. Only the selected submission received a private score. Its 0.3652 historical result is not an estimate of how it would have performed on those 52 private references.

The Japanese rubric missed one private example. With only 35 private rows, one error moves accuracy by about 2.86 percentage points. Without the private row identities and labels, we do not know what caused that error. I would want to test the rules on a much larger and less templated set of responses.

The research I want to continue

Across the three tasks, we kept running into the problem of choosing between candidate predictions. In Greek, we made progress by accepting a correction only when several views supported the same operation. In Japanese, diverse models contained complementary correct predictions, but our selectors could not reliably identify them. In QA, a small collection of grounded edits contained better answers on many rows, yet the benefit did not translate into a selector that generalized across issuers and periods.

I have little time for this at the moment. If time allows, or if there is a follow-up competition with a harder private test, I would continue the research in these directions:

Greek extraction: make the correction policy transfer across document sources and annotation versions, while retaining the ability to decline an uncertain edit.
Japanese intent: model the question’s event, the response’s commitment to that event, and the scope of conditions and negation, with substantially better data for rare refusals.
Financial QA: combine verified facts with a better account of answer scope, and treat a retrieved historical answer as a candidate whose claims and version need checking.

For benchmark design, our experience also raises practical questions. How should a shared task handle publicly available answers to reused inputs? Can results separate exact-overlap cases from genuinely new reports? When references change, can their versions be documented alongside the inputs? And how should we report a system developed after inspecting the test packet, even when no hidden labels were available?

Khaled and I are pleased with the rankings. We would be much more interested in demonstrating that the systems we built can handle new filings, indirect answers and unfamiliar financial questions.

If you are at EMNLP or FinNLP in Budapest, we would be happy to discuss the experiments, including the failures.

The three system papers provide the detailed methods and reproducibility information: