AI Architectures

While training data is important for the final functionality of an AI model, it is not the only consideration. In order for that data to end up being useful an appropriate architecture to base the model on is crucial. Think of the architecture as a skeleton, just a bunch of bones which need to be connected by muscles. There are many types of architecture that can be used for different purposes.

Some AI models are narrow, known as artificial narrow intelligence (ANI) and are optimized for one very specific purpose like chess AIs, while others are called artificial general intelligence (AGI) and attempt to mimic human understanding in all fields. AGI has not been achieved yet and there are many debates about definitions and consciousness.

Large language models (LLMs) like ChatGPT are an attempt at creating AGI and are significantly more advanced than the technology was a few years ago. That being said, for either category there are many different architectures, each with their superpowers, limitations, and specific use cases for which these tradeoffs make sense. To give a basic understanding of what is possible we will only discuss 2 of these – Genetic Algorithms and Neural Networks.

Genetic algorithms (GAs) are one type of AI architecture, a little different than what you are probably used to seeing in the mainstream. In order to widen understanding of different types of AI we will look into what they are useful for and how they work. This is a very specialized type of architecture that is not broadly useful for many different types of tasks but is highly efficient for particular problems.

Usually GAs are used in optimization problems where there are more variables than what can be brute forced. It is an elegant lightweight solution for testing the possible solutions to strictly defined problems. All of this may sound a little abstract so let’s investigate the most popular example – the traveling salesman problem.

In the drawing above you can see 7 nodes. The traveling salesman problem expects a solution where every node or city is visited exactly once, and the goal is to find the path that is shortest. For simplicity there are no distances between the nodes in this example. One potential solution is ABCDEFG. Another solution is ABCDEGF. While there are other ways to solve this, GAs perform really well in such problems.

So how do they work? In this example, every solution is 7 letters long and includes each letter exactly once. Think of this as the DNA on which the genetics are based, the location of each letter is an attribute of the solution. Each solution is a member of the population.

Initially, a random population is generated with all types of solutions, some of which are inefficient and others are faster. A fitness function is a ranking system used to compare the quality of solutions. In this example the fitness function is just the distance of the path with lower distance being better. This means that lower total distance solutions will be ranked better and be used to produce the next generation. With 3 nodes if the solution is ABC, the distance is AB + AC, where AB is the distance from A to B.

Once the population is generated, it should be sorted on the fitness function. Then, the next generation can be produced from parents of the current generation. Although higher ranked solutions are more likely to reproduce, there is some randomness to avoid premature convergence and local maxima. This topic is too nuanced for a deep dive within the scope of this course but extra resources can be found for curious learners.

Reproducing is done by taking the 2 parents and combining them to vary the offspring. While there are many different ways to combine the genetics, the general idea is that the new population is on average better since the least effective solutions from each generation are removed. By running the algorithm for multiple generations you can keep track of the best solutions until one is good enough for the purpose as there is never a guarantee that it is the best possible solution.

Neural networks (NNs) are the cornerstone of modern AI. The concept is that they mimic how human brains work with neurons (brain cells) and the connections between them. In the context of AI each neuron is called a node and its computer representation is a function for processing data. One simple case is where the node has a value between 1 and 0 and the input is multiplied by the value, this is a function. The connections also frequently have values called weights, representing the influence of one node on the next.

There are a vast number of different structures for NNs, including the transformer architecture – this is what most LLMs like ChatGPT are based on. In this chapter we will discuss the broad idea of a neural net rather than the details that make up more advanced architectures.

To develop an intuition for how NNs work we will use a visual example of an image recognition AI.

In this example the nodes are the circles and the arrows are the connections. The weights of the connections and the nodes both impact the results. We will understand where these values come from in chapter 4 – training AI.

Now imagine a grid like this over an image. Instead of these squares, however, each pixel is a blue input circle in the NN. These inputs are then processed by the network, each connection analysing patterns between the neighbouring pixels. When the input passes through enough hidden layers it is capable of identifying structures based on the input pixels. It would then be able to describe what these structures are.

The capabilities are dependent on the size of the neural net as it can only take as many inputs as it has nodes. Think of it as a puzzle: if each node could tell you where a piece goes, you would need as many nodes as you have pieces. This is oversimplified though and in reality it is beneficial to have more nodes for accuracy and nuance. While we learned that training data is important in the previous chapter, the size of the NN is also a bottleneck for performance. If you had one node, no matter how much data you fed it and how good your data is, all you would do is update one value and in the end your AI would not be useful.

With the understanding from this chapter it should be clear that the architecture of an AI model will directly impact what it is capable of. Neural networks are more versatile as the input data can be flexible if the size of the brain (number of parameters) is sufficient. Genetic algorithms are use case specific due to the enforced input and output, but certainly have their strengths when used in the right context. There are far more architectures to be explored, some very advanced and some more basic. In general basic models are faster to run and preferred for well defined expectations. Advanced models are expensive to train and require state-of-the-art hardware to run, but produce mindblowing results if engineered effectively. Always make sure to analyse which architecture is suitable as it will determine the results. Stockfish is a good chess AI, better than any human in history, while ChatGPT still makes illegal moves throughout its games. Both are AI but they are trained with different intentions.