Distributed and Democratized Learning Philosophy

This is based on our medium post.

The full detail of the paper can be found on ArXiv.

We develop a novel design philosophy called democratized learning (Dem-AI) whose goal is to build large-scale distributed learning systems that rely on the hierarchical self-organization of distributed learning agents that are well-connected, but limited in learning capabilities. Dem-AI learning system can evolve and regulate itself based on the underlying duality of two processes that we call specialized and generalized processes.

Artificial Intelligence (AI) is moving towards edge devices with the availability of massively distributed data sources and the increase in computing power for handheld and wireless devices such as smartphones or self-driving cars. This has generated growing interest to develop large-scale distributed machine learning paradigms.

Democracy in Learning

Democracy in learning features a unique characterization of participation in the learning process, and consequently develops the notion of democracy in learning whose principles include the following:

• According to the differences in characteristics of learning agents, they are divided into appropriate groups that can be specialized for the learning tasks. These specialized groups are self-organized in a hierarchical structure to mediate voluntary contributions from all members in the collaborative learning for solving multiple complex tasks.
• The shared generalized learning knowledge supports specialized groups and learning agents to improve their learning performance by reducing individual biases during participation. In particular, the learning system allows new group members to: a) speed up their learning process with the existing group knowledge and b) incorporate their new learning knowledge in expanding the generalization capability of the whole group.

To that end, these characteristics motivate us to develop a novel design philosophy for future large-scale distributed learning systems.

In summary

Existing machine learning designs (such as meta-learning, multi-task learning, reinforcement learning, federated learning) face critical challenges to scale up the current centralized AI systems into the distributed AI systems that can perform multiple complex learning tasks. Therefore, in the future, a transition from traditional centralized learning systems towards large-scale distributed AI systems is imperative.

Takeaway

We realize that Dem-AI philosophy is relatable in our development process, for instance, in dealing with representation and diversity when doing collaborative tasks, and further, reducing biases through open discussion. Most importantly, Dem-AI philosophy enriches our understanding of a distributed learning system with a broader perspective, and in practice, it opens up unlimited applicabilities for future distributed learning systems.