DeepSeek’s New Architecture Can Make AI Model Training More Efficient and Reliable

DeepSeek, the Chinese artificial intelligence (AI) startup, that took the Silicon Valley by storm in November 2024 with its R1 AI model has now revealed a new architecture that can help bring down the cost and time taken to train large language models (LLMs). A new research paper has been published by the company outlining a training architecture called Manifold-Constrained Hyper-Connections (mHC), aimed at improving the efficiency and reliability of large AI model training. It is focused on reducing instability during training runs, a challenge that can lead to wasted compute resources and interrupted training progress.

DeepSeek Brings New AI Training Architecture

In a paper published in arXiv and listed on Hugging Face, DeepSeek researchers introduced and detailed the new model training architecture. The mHC architecture is a structural tweak to neural network layers that constrains how information flows across the model during training. Existing frontier models often use pathways that let data bypass some processing steps to keep the signals stable across multiple layers. However, expanding these shortcut paths without any constraints can introduce instability and make large models harder to train end-to-end.

The new architecture proposes a change to fix this issue. With mHC, researchers project these connections onto a specific structured space called a manifold, which mathematically ensures the signals remain stable while passing through layers.

Simply put, large AI models use billions of parameters or neural connections, with each of them impacting the pattern and behaviour of the end result. This is why response to the same query on ChatGPT differs slightly on Gemini or Claude. Training a model essentially requires users to adjust every single parameter to get a desired result.

During this process, if signals (the data passing through different parameters) are projected strongly or vanish quickly, the training can fail halfway through the process forcing developers to restart. This can waste time, money, and precious compute power. mHC's design tries to curb this behaviour by keeping the shortcuts in the model's computation predictable and well-behaved.

DeepSeek's research team tested the new architecture of multiple model sizes, including a 27 billion-parameter model trained on data proportional to its scale, as well as smaller variants. This was done to study how compute and dataset size interact with the architecture. The team found that mHC helps even large AI models maintain stability and scalability without excessive overhead.

The practical goal of mHC is not only to improve stability but also to reduce the wasted costs associated with interrupted training runs. Training large AI models can require substantial energy, specialised chips and long runtimes. DeepSeek's approach does not directly lower the power draw of hardware like GPUs or AI accelerators, but by reducing the frequency of training failures and the need to restart, it can lower the total compute consumed across a training lifecycle.

Since the architecture is currently not part of any market-ready AI models, it is difficult to gauge how it will behave when stress-tested in real-world scenarios. However, on paper, it does offer an alternative compared to the existing techniques, and can be a fundamentally better way to train AI models. We will have to wait until independent researchers incorporate the training architecture in their models and share results, or the paper is peer reviewed and scrutinised.