📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva project trained a large-scale, native-language LLM from scratch but achieved surprisingly low performance on Italian academic tests. This challenges assumptions about the scale needed for effective country-specific models.
Italy’s Minerva-3B, a large-scale language model trained from scratch on 2.5 trillion tokens with roughly half Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, highlighting a significant challenge for sovereign-language AI projects.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s national supercomputing center CINECA and funding through Italy’s national AI strategy. The project trained models ranging from 350 million to 7 billion parameters, with the 3B model being the focus of recent evaluations.
Despite the large-scale, native-language training, Minerva-3B’s performance on the INVALSI benchmark was near chance, at just 4.9%. Researchers noted that while dataset composition is important, overall dataset size and model parameters are more critical for complex language tasks, suggesting that the current scale may still be insufficient for high-level academic comprehension.
This empirical result complicates the narrative that simply increasing native-language data and parameters guarantees deep country-specific knowledge in LLMs, raising questions about optimal investment levels and scale.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Build a Large Language Model (From Scratch)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Fine-Tuning Large Language Models: From Custom Datasets to High-Performance AI Models Using Modern Toolchains
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 (up to 5.1GHz) Mini Gaming Computers, 128GB LPDDR5X 8000MHz (16GB*8) 2TB PCIe 4.0 SSD, Quad Screen 8K Display, WiFi 7 & USB4, SD Card Reader 4.0
EVOLUTION RYZEN AI MAX+ 395 MINI PC – GMKtec EVO-X2 is the next evolution in AI mini PC…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Your Personal AI Supercomputer: Build Your Own AI Workforce
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign-Language AI Strategies
The low performance of Minerva-3B despite substantial investment and large-scale training indicates that achieving meaningful country-specific AI capabilities may require even greater resource commitments. This challenges the assumption that training from scratch on native data alone is sufficient and suggests that European projects may need to reevaluate their scaling strategies to justify national investments.
It also underscores the importance of empirical benchmarks in assessing progress, as high-quality models must perform well on real-world, academically relevant tasks to be truly effective. The findings could influence future funding, research priorities, and the design of sovereign-language AI initiatives across Europe.
Background on European Sovereign-Language Models and Minerva Development
Italy’s Minerva project emerged as a counterpoint to the European debate over whether to develop LLMs from scratch or extend multilingual models through continuation pre-training. Unlike Portugal’s AMÁLIA, which layered European Portuguese onto a multilingual foundation, Italy built Minerva from scratch, training on 2.5 trillion tokens, with approximately 50% Italian content. The project was supported by Italy’s national research infrastructure, including CINECA’s supercomputers, and aimed to demonstrate the viability of a fully sovereign, native-language AI model.
Previous efforts in Europe have often focused on smaller-scale models or continuation training, with the assumption that native-language specialization would yield superior results. However, Minerva’s recent benchmark results challenge this view, revealing that scale and dataset size may be more critical than previously thought, especially for complex language understanding.
“Our evaluation indicates that dataset size and parameter count are more influential than dataset composition alone. More investment may be necessary to reach desired performance levels.”
— Research team member, Minerva project
Unresolved Questions About Scale and Effectiveness
It remains unclear whether increasing the size of training data and model parameters beyond current levels will significantly improve Minerva’s performance on academic and complex language tasks. The results suggest a potential need for even larger models or different training methodologies, but definitive conclusions are still pending.
Additionally, how these findings translate to other European languages and projects is not yet known, and ongoing research is needed to determine optimal investment levels for sovereign-language AI development.
Next Steps for European Sovereign-Language AI Development
The Minerva team is continuing to iterate on their models, including upcoming evaluations of larger models and refined training strategies. Researchers will likely investigate whether scaling further or adopting new methodologies can bridge the performance gap highlighted by recent benchmarks.
European policymakers and research institutions may reassess funding priorities and collaboration strategies based on these empirical findings, potentially emphasizing larger-scale investments or alternative approaches to achieve meaningful country-specific AI capabilities.
Key Questions
Why did Minerva perform so poorly on the Italian INVALSI benchmark?
Despite being trained on a large, native-language dataset, Minerva-3B’s performance was limited by the scale of the model and dataset size relative to the complexity of academic language tasks. The findings suggest that current scale levels may still be insufficient for deep understanding.
Does this mean training from scratch is ineffective for sovereign-language models?
Not necessarily. The results indicate that scale and dataset size are critical factors. Training from scratch can be effective if sufficient resources are allocated, but smaller-scale efforts may not achieve desired performance levels.
What implications does this have for other European countries developing sovereign LLMs?
It suggests that similar projects should consider scaling up their models and datasets significantly, and that empirical benchmarks are essential for assessing progress and justifying investments.
Will increasing model size improve performance on academic tasks?
Likely, but it remains an open question. Ongoing research by Minerva and others will test whether further scaling can overcome current limitations.
How does Minerva compare to multilingual models like AMÁLIA?
Minerva was trained from scratch on predominantly Italian data, whereas AMÁLIA extended multilingual models with smaller amounts of European Portuguese data. Minerva’s results demonstrate that larger scale does not automatically guarantee better performance, raising questions about the optimal approach for sovereign-language models.
Source: ThorstenMeyerAI.com
