Proposed
Proposed Tracks
From prototypes that answer questions to systems that collect evidence, measure results and make changes explainable. The three tracks are independent but share one principle: every insight must be verifiable.
This is the first set of proposed tracks. They may be refined before the event.
Registration closes on October 31 (23:59, Rome time).
Register a team and share its secret code, or join a team with a code from a teammate. On your own? Register as looking for a team. Registering a team gives priority for participation confirmation.
How teams workTrack 01
Research Radar
Mapping the Evolution of Scientific Knowledge
- Guiding question
- How is a research field changing?
- Main result
- A temporal map of the literature
Mission
Given a scientific topic and a time range, build a system that ingests publications from open sources, identifies the main strands of work, reconstructs the relationships between papers and flags which directions are accelerating or slowing down.
The problem
Research evolves too fast to follow by hand. Keyword searches return lists of papers, but rarely show reliably how new subtopics emerge, which approaches converge and which results actually changed the landscape.
Who it is for
Researchers, R&D teams, deep-tech investors, innovation teams.
- Input
- A topic (e.g. reinforcement learning), a main period (e.g. 24 months), a comparison window (e.g. the last 14 days), and optional sources and filters.
- Output
- Topic clusters, a timeline, a paper–citation graph, a summary of recent movements, representative papers and links to the sources.
- Distinctive element
- A “What changed?” mode that compares two homogeneous time windows and also makes explicit any variation caused by source coverage.
A user selects “reinforcement learning”, from October 2024 to October 2026, and asks what happened in the last 90 days compared with the previous 90. The system retrieves metadata and abstracts, proposes clusters such as offline RL, model-based RL and RL for agents, and computes the relative growth of publications per cluster. A summary flags the growing clusters and shows 3–5 representative papers with a rationale and links. Cluster names are hypotheses generated from the data, not predefined labels.
- Ingestion from arXiv, OpenAlex and, for biomedical topics, PubMed/Europe PMC, through permitted APIs and downloads; deduplication by DOI/arXiv ID and title.
- Normalisation of title, authors, date, abstract, venue, identifiers, citations and full-text availability.
- Semantic clustering with interpretable labels, and the ability to correct incoherent clusters.
- Relationships between papers: documented citations, semantic similarity and shared authors, shown as distinct edge types.
- Temporal analysis: absolute volumes, share per topic, acceleration, novelty and comparison between windows.
- A report with verifiable claims, precise links to the sources and an indication of uncertainty.
MVP
- One domain and one source, with 500–2,000 metadata/abstract records and a reproducible query.
- 5–12 clusters with readable labels, example papers and a timeline.
- A comparison of two time windows, with at least three insights backed by sources.
- A mini-dashboard with filters by period and cluster, and access to the original papers.
Stretch goals
- Cross-source deduplication
- Interactive citation graph
- Novelty score
- Recurring alerts
- Comparison between authors and labs
- Grounded conversational search
- Full-text extraction where permitted
Retrieval quality on an annotated sample; cluster coherence assessed manually; precision of the relationships; correctness of dates; share of report claims with a valid source; usefulness of the trends against simple baselines. Note: growth in the number of papers does not automatically mean higher quality or impact.
These are indicative pointers for this track. The final judging criteria will be announced separately.
Track 02
GitHub Benchmark Arena
Continuous Evaluation of Emerging Open-Source Tools
- Guiding question
- Which new tools actually work?
- Main result
- A continuous benchmark and a leaderboard
Mission
Given a software topic, find relevant open-source repositories, install and run them in a controlled environment, and compare them on shared tasks with reproducible metrics and a historical leaderboard.
The problem
New frameworks and libraries appear every week, with claims that are hard to compare. GitHub stars and promotional demos do not measure reliability or performance on real workloads.
Who it is for
Developers, platform engineers, AI infrastructure teams, CTOs and applied researchers.
- Input
- A category (e.g. agent harness), a GitHub query, minimum requirements, a task suite and a runtime budget.
- Output
- Eligible repositories, installation outcomes, benchmark results, metrics, versions, logs and a leaderboard.
- Distinctive element
- Incremental re-runs when a repository changes or a new candidate enters, keeping experiments comparable.
The platform selects 3–5 harnesses compatible with the same model and the same tools. Each harness runs 20 tasks, for example searching a small knowledge base, editing test files and solving problems with mock tools. Success rate, duration, number of calls, tokens and estimated cost are measured. For open-ended text outputs, an LLM-as-a-Judge applies a fixed rubric; tasks with an exact answer are scored deterministically.
- GitHub discovery by topic, licence, recent updates, language and compatibility.
- A standard adapter that maps each repository to one execution interface.
- Automated setup in ephemeral containers with pinned dependencies, timeouts, CPU/RAM limits and a blocked network or an allowlist.
- A gold-standard benchmark with inputs, expected outputs or success criteria, edge cases and scoring.
- Metrics: task success, p50/p95 latency, memory use, error rate, cost and stability across runs.
- An LLM judge with rubrics, calibrated examples, tracked judgements and human review of a sample.
- A leaderboard filterable by version, date, task and quality/cost trade-off.
MVP
- Manually select 3 compatible open-source repositories, then automate discovery of new releases.
- Define 10–20 tasks and a common adapter, with an explicit baseline.
- Run each tool at least twice, saving the commit SHA, configuration and logs.
- Show the leaderboard, a comparison by metric and at least one reproducible failure case.
Stretch goals
- Fully automatic discovery
- Multi-model benchmarks
- Statistical significance and confidence intervals
- Regressions between releases
- Benchmark contamination detection
- Periodic scheduling and notifications
Do not run unknown repositories on the host: use least-privilege sandboxes, no secrets and a restricted filesystem. Check licences, hardware requirements, compatibility and API policies. An LLM judge is not a gold standard by itself: it must be calibrated with annotated examples and backed by deterministic checks.
Reproducibility of runs; correctness of the adapters; coverage and quality of the benchmark; transparency of scoring; robustness to setup errors; quality of the leaderboard; execution security.
These are indicative pointers for this track. The final judging criteria will be announced separately.
Track 03
Agent Failure Lab
Understanding Why AI Agents Fail
- Guiding question
- Why does an agent fail, and how can it be improved?
- Main result
- Diagnosis, replay and verification of fixes
Mission
Observe the execution traces of AI agents, recognise and classify failures, identify the most plausible cause and propose changes that are tested with new runs and regression benchmarks.
The problem
When an agent fails, the final result rarely explains whether the cause was a wrong plan, an incomplete search, a misused tool or a loop. Analysing long traces by hand is slow and does not scale.
Who it is for
Teams building agents, QA, MLOps, AI product engineering.
- Input
- A task, an expected outcome or success test, a structured trace (messages, tool calls, outputs, timings, costs) and the agent configuration.
- Output
- A failure taxonomy, suspicious steps, an explanation with evidence, severity, a proposed fix and a before/after comparison.
- Distinctive element
- A fix is not considered valid until it improves a test and causes no obvious regressions.
Task: “Fix a parsing function and make the tests pass”. The agent opens the wrong file, edits an unrelated module and concludes the task is solved without running pytest. The platform reconstructs the sequence, flags a possible tool-selection failure and a verification failure, cites the trace steps and suggests a mandatory test check before the final message. The replay verifies whether the new policy increases the number of solved tasks without raising cost and latency excessively.
- Trace ingestion in JSON with a common schema: timestamp, step ID, role, tool, input/output, status, and cost and tokens where available.
- Visual replay of the sequence, highlighting the steps that precede the failure.
- A hybrid failure detector: deterministic rules (timeouts, loops, failed tests) plus LLM classification with a rubric.
- Root-cause hypotheses citing specific steps, with a confidence level, distinguishing symptom from cause.
- A remediation engine that proposes prompt patches, guardrails, tool validation or strategy changes.
- Before/after evaluation on a fixed task set, with regressions and costs made explicit.
Initial failure taxonomy
- Planning: incomplete plan, neglected dependencies, wrong priorities.
- Tool use: wrong tool, malformed parameters, unhandled errors.
- Retrieval/context: insufficient sources, irrelevant information, lost context.
- Reasoning/decision: unsupported inferences, premature choices, repeated attempts.
- Execution: timeouts, crashes, resource limits, environment incompatibilities.
- Verification: output not checked, tests not run, false claim of success.
MVP
- Collect 15–30 traces, even synthetic but realistic ones, with at least 4 failure categories and annotated ground truth.
- Build a parser, a timeline and a failure classifier with per-step evidence.
- Implement two concrete remediations, for example a controlled retry and a verification gate.
- Run a replay on 5–10 tasks and measure success rate, regressions, latency and cost before and after.
Stretch goals
- Live trace ingestion
- Causal debugging through controlled interventions
- Clustering of recurring failures
- Automatic generation of regression tests
- Patch suggestions with human approval
- Comparison across agents and versions
Classification accuracy on annotated traces; precision in locating the relevant step; quality and verifiability of explanations; rate of effective fixes; regressions introduced; additional cost. Avoid presenting an LLM diagnosis as causal proof without an experiment.
These are indicative pointers for this track. The final judging criteria will be announced separately.
Shared principles
Reproducibility
Sources, versions, parameters and results must be traced.
Evidence first
Separate objective measurements, LLM assessments and interpretations.
Human in the loop
Allow review of the most uncertain results.
Security by design
Respect licences, privacy, rate limits and the isolation of external code.
Realistic scope
A robust end-to-end pipeline on a narrow use case is better than a broad but fragile demo.
What to deliver
- A GitHub repository with a README and run instructions
- A 5-minute demo
- Datasets or test fixtures
- Quantitative results
- A description of the limitations
- A short roadmap
A UI is recommended but not mandatory if a CLI or API can be demonstrated.
Sources, stack and numbers mentioned in the tracks are design suggestions, not rigid requirements.
Judging criteria, prizes and jury
To be announced. We will publish them on the home page and on Luma.