AlphaFold Decoded: iGEM Education Initiative

This lesson introduced the concept of tensors, explaining that they are n-dimensional arrays crucial for representing data in deep learning models like AlphaFold. Tensors generalize familiar structures like vectors and matrices, allowing us to work with multi-dimensional data efficiently. In PyTorch, tensors are the core data structure, and the framework offers efficient tensor operations and automatic differentiation to facilitate model training. We began by learning how to create tensors in PyTorch using functions like torch.tensor, torch.linspace, and torch.ones, and explored how tensors support mathematical and logical operations. Understanding tensor shapes and manipulation, including reshaping and indexing, is key to working with multi-dimensional data.
The lesson also introduced axis-wise operations, where computations are performed along specific dimensions of a tensor. Operations like torch.sum, torch.argmax, and torch.mean were explained in the context of summing or reducing along axes. Non-reducing operations like the softmax function, which is vital for generating probability distributions, were also covered. We discussed broadcasting, a powerful technique that allows operations on tensors with mismatched shapes by implicitly expanding smaller tensors. The lesson demonstrated how broadcasting simplifies computations, such as adding a vector to each row of a matrix, and introduced the ellipsis operator for handling higher-dimensional tensors.
Finally, the torch.einsum function was introduced as a concise and flexible way to perform complex tensor operations like matrix multiplication and the outer product. This function uses Einstein notation to handle operations across multiple dimensions efficiently. Throughout the lesson, we emphasized that these tensor operations form the foundation for more complex tasks in AlphaFold, from handling input data to computing model predictions. By mastering tensor creation, manipulation, and advanced operations, learners will be well-prepared to tackle the upcoming challenges in protein structure prediction.

This lesson introduced machine learning concepts, focusing on building a two-layer feed-forward neural network to classify handwritten digits. Starting with the idea of pattern recognition through dot products, we demonstrated how a machine learning model identifies hierarchical features, such as specific image parts like circles or lines. The network processes images by flattening them into vectors, which are multiplied by weight matrices that represent feature templates, followed by matrix multiplications to derive predictions. To prevent collapsing into a single matrix multiplication, non-linear activation functions like ReLU are applied between layers.
We discussed the importance of using biases in linear layers to improve flexibility and model accuracy. Next, we covered the loss function, focusing on L2-loss, which measures the difference between predicted outputs and actual labels. We explained how the model optimizes its weights using gradient descent, where gradients—calculated via backpropagation—inform the direction and magnitude of parameter updates to minimize the loss.
We then delved into the math of calculating derivatives for various layers, from the L2-loss and sigmoid to ReLU and affine linear layers. While the gradient computation can be complex, PyTorch handles these calculations automatically through its autograd feature. The overall training process involves multiple iterations of forward passes, loss computation, gradient calculation, and weight updates. Finally, we emphasized how this foundational machine learning process prepares us for the more advanced models, including AlphaFold. The lesson provided hands-on learning with a tutorial notebook to apply these concepts and create a machine learning model from scratch.

This lesson provided an in-depth introduction to attention in machine learning, starting with its core idea of focusing on relevant parts of an input sequence. Attention uses key, query, and value vectors, which are created from the input data, to compute attention scores based on their dot-product similarities. These scores, normalized with softmax, help the model decide which inputs to prioritize when making predictions. This process allows for parallel processing of sequences and makes attention an ideal tool for tasks involving language, music, or protein sequences, like in AlphaFold.
We discussed several methods for applying attention to machine learning models, including self-attention, where both the queries and keys come from the same inputs. Self-attention is particularly useful in tasks like protein structure prediction or natural language processing, as it helps models understand the relationships between different parts of a sequence. We also introduced multi-head attention, which runs several attention mechanisms in parallel to capture more diverse aspects of the input.
One of the practical challenges of attention is its high memory usage due to the N² scaling of attention scores. AlphaFold addresses this with techniques like global attention, which reduces memory requirements by averaging the query vectors. Another challenge is handling sequential data, especially in tasks like next-word prediction, which requires causal attention masks to prevent the model from "cheating" by looking at future tokens. These masks ensure the model only attends to past tokens during inference.
We also explored the issue of permutation invariance in attention models, where the order of input tokens can be ignored by the model. To solve this, positional encodings are added to the input, allowing the model to learn the sequence structure. These encodings can be either static (like sinusoidal functions) or learned, both of which are used effectively in models like AlphaFold.
The lesson highlighted the transformative impact of attention in machine learning, as seen in the widely-used Transformer architecture. Transformers, which rely on attention, revolutionized natural language processing by enabling faster and more efficient sequence-to-sequence tasks like translation. We covered the key components of the Transformer, including multi-head attention, residual connections, and feed-forward layers, which together form the foundation of modern AI models.
Finally, we emphasized the importance of attention in AlphaFold, where it plays a crucial role in processing the sequences of amino acids to predict protein structures. With this foundational understanding of attention, the next steps in our series will focus on implementing these concepts in AlphaFold, starting with feature extraction from protein data files. The tutorial notebook for this lesson will help reinforce these ideas by guiding you through building a multi-head attention module and a Transformer encoder for sequence classification tasks.

In this lesson, we dive into AlphaFold’s feature extraction process, focusing on converting biological data into tensors for machine learning. AlphaFold uses three primary inputs: the amino acid sequence, MSA (multiple sequence alignment) data, and 3D structures of templates, though we simplify our approach by omitting the template stack. The amino acid sequence is straightforwardly converted to a one-hot encoding, but MSA data requires more complex processing due to variations like insertions, deletions, and evolutionary mutations across sequences. We introduce the Needlemann-Wunsch algorithm for aligning sequences and explain how deletions are handled by counting them while disregarding specific deleted amino acids.
The MSA data is processed into several key features, such as amino acid profiles and deletion counts. Unique sequences are clustered, and cluster centers are chosen, always including the target sequence. These clusters undergo a masking process where amino acids are randomly replaced to enhance model robustness. After masking, extra sequences are assigned to the clusters based on their similarity to the cluster centers, and features like deletion counts and amino acid types are averaged across clusters.
We then calculate the features for cluster centers and extra sequences, which are used as inputs to AlphaFold’s Evoformer module. These features include the one-hot encoding of amino acids, deletion counts normalized by an arctan function, and amino acid profiles averaged over clusters. The process involves randomness in selecting cluster centers and during masking, ensuring varied inputs across multiple prediction runs. Finally, we concatenate the features to create inputs like msa_feat and extra_msa_feat, ready for AlphaFold’s further processing.
This lesson emphasizes how feature extraction in AlphaFold leverages evolutionary data to inform protein structure prediction. It also highlights the importance of domain-specific data transformation for effective machine learning. In the next videos, we’ll explore how AlphaFold’s Evoformer module transforms these features into predictions, and later, how it outputs protein structures from these tensor inputs.

This lesson delves into AlphaFold’s Evoformer, the largest component of the model, accounting for 88M of the 93M parameters. The Evoformer plays a key role in transforming the MSA (multiple sequence alignment) and pair representations into features useful for protein structure prediction. It consists of multiple identical blocks that include an MSA stack (working on the MSA representation) and a pair stack (working on pairwise residue interactions). The main challenge of processing large proteins is tackled by employing row-wise and column-wise attention mechanisms instead of full attention, which would be computationally expensive.
The MSA stack performs row-wise attention using the pair representation as a bias, followed by column-wise attention and a feed-forward transition module. The pair stack features a unique triangle attention mechanism, which incorporates information from other residue pairs to ensure the structural consistency of the protein. Specifically, the triangle attention computes updates between residues using information from their shared neighbors, mimicking the geometric relationships in a protein's 3D structure.
The Evoformer block has an “Outer Product Mean” operation, which reshapes the MSA representation into the pair representation, facilitating information exchange between the two. The pair stack’s attention mechanisms, triangle multiplicative updates, and transition modules help refine the representation of pairwise interactions to generate the final prediction. Residual connections throughout the Evoformer ensure stable training, allowing the model to add learned corrections rather than replace inputs completely.
Finally, the MSA representation is reduced to a single sequence, the target sequence, which is passed on to the structure module for further refinement. The Evoformer, though more complex than a traditional transformer, is composed of straightforward operations that handle the enormous protein data by simplifying attention mechanisms. The next video will explore the missing input embedding step, linking the extracted features to the Evoformer’s processing pipeline, while this lesson's notebook provides a hands-on guide to implementing the Evoformer.

This video focuses on feature embedding in AlphaFold, which converts extracted data into a format suitable for the Evoformer. Feature embedding is done in three main steps: the input embedder, the Recycling Embedder, and the Extra MSA Stack. The input embedder processes the target and MSA features, embedding them with linear layers and adding positional encodings. The positional encodings are derived from residue indices, and relative positions are one-hot encoded to ensure the model can understand residue relationships.
The Recycling Embedder, used during iterative predictions, updates the pair and MSA representations using the predicted pseudo-beta carbon positions from the last iteration. By calculating pairwise distances between these carbons, encoding them, and normalizing the representations, the Recycling Embedder integrates previously predicted structural information into the next prediction. During training, AlphaFold uses this recycling approach to improve efficiency by only optimizing parameters for the final prediction, skipping backpropagation for intermediate steps.
The Extra MSA Stack updates the extra MSA and pair representations and is almost identical to the Evoformer but uses fewer blocks (4 instead of 48) and includes global attention for better memory efficiency. Global attention reduces the computational load by averaging queries and broadcasting outputs, making it suitable for large datasets like the extra MSA sequences. The lesson wraps up by introducing key concepts like global attention and recycling while setting the stage for the upcoming construction of the structure module.
The next video will dive into 3D geometry concepts like rotation matrices and quaternions, which are essential for building the protein’s 3D structure, marking a deeper dive into the final parts of the AlphaFold model.

In this video, we discuss the advanced 3D geometry concepts required to implement AlphaFold’s Structure Module, focusing on how to transform predicted tensor outputs into atomic positions. AlphaFold predicts three scalar coordinates for backbone translations and a quaternion for backbone orientation. Quaternions, rather than rotation matrices, are used for rotations because they are better suited to neural networks and maintain uniform distributions of rotations. Quaternions are scaled to a unit length, which allows a uniform distribution of rotation angles around an axis.
For torsion angles, AlphaFold predicts two values (cosine and sine), representing bond rotations, and these angles are applied to the side-chain atoms. The video explains how transformations are composed of a rotation and a translation, modeled using 4x4 matrices in homogeneous coordinates. This allows efficient application of rotations and translations as matrix multiplications, which is crucial for handling complex protein structures. The concept of transforming local frames to global frames is key to understanding how atomic positions are constructed in AlphaFold.
The backbone frame is defined using the backbone atoms, such as the C-alpha, nitrogen, and carbonyl groups, while side-chain frames are defined relative to these. We use transformations to update the frames as we apply the torsion angles for different residues. Additionally, the video explains how to use the Gram-Schmidt process to orthonormalize vectors for constructing rotation matrices, ensuring that rotations are well-behaved in 3D space.
After constructing the rotation and translation matrices, we apply these to calculate global frames for each amino acid. Finally, these global frames are used to compute the exact 3D positions of atoms within the protein. With the foundation of these geometric transformations, AlphaFold can build the entire protein structure from the predicted backbone and side-chain transformations. The accompanying Jupyter notebook provides hands-on implementation of these concepts, bridging the gap between theoretical geometry and practical AlphaFold development.

In this video, we explore the Structure Module of AlphaFold, which takes the Evoformer outputs and converts them into actual atom positions. A core part of this module is Invariant Point Attention (IPA), which replaces standard attention with a mechanism that computes attention weights based on the 3D distance between residues, treating key and query vectors as points in space. These distances are calculated using the current residue positions and transforms, updated iteratively across several rounds. Instead of only considering dot-product similarities for attention, AlphaFold integrates spatial proximity by globalizing predicted local coordinates and weighting closer residues more heavily.
Each iteration of the Structure Module refines the backbone’s rotation and translation using quaternion-based updates, gradually improving the protein’s predicted structure. AlphaFold refines the backbone transforms in eight iterations, initializing all residues at the origin in the first round (referred to as "black hole initialization"). Along with updating positions, the module computes torsion angles for the protein backbone and side chains. During training, predictions from each iteration contribute to the final loss, ensuring the model can produce accurate outputs at each step.
The Backbone Update module predicts a quaternion and translation for each residue. AlphaFold uses a simplified method of predicting quaternions, initially setting the scalar part to one, to ensure small, incremental rotations across iterations. After calculating the final backbone transforms and torsion angles, the method computeAllAtomCoordinates from the previous video is used to compute the exact atom positions.
The Structure Module outputs the torsion angles, backbone transforms, heavy atom positions, atom masks, and pseudo-beta positions. The pseudo-beta positions are used for recycling into subsequent AlphaFold iterations. AlphaFold also manages unit conversion between nanometers and angstroms, scaling the translation vectors accordingly. With this, the Structure Module completes the protein structure prediction, and in the next video, we'll combine all parts for the full AlphaFold implementation.

In this final video, we put together the entire AlphaFold model from the modules we've built over the series. The model is composed of six main modules: Input Embedder, Extra MSA Embedder, Recycling Embedder, Evoformer Stack, Extra MSA Stack, and the Structure Module. These modules process inputs, create embeddings, and refine representations to predict accurate atom positions for protein structures. The process starts with Input Embedding, which creates the MSA and pair representations. The Recycling Embedder adjusts these representations using outputs from prior model iterations, particularly based on pseudo-beta carbon distances.
The Evoformer Stack, a transformer-like module, is the core of the network, performing updates through row-wise and column-wise attention layers. The Extra MSA Stack processes sequences not selected as cluster centers, contributing via a shallower model. After Evoformer, the Structure Module transforms the enriched tensors into precise 3D atom positions, using Invariant Point Attention based on residue proximity.
The full AlphaFold Inference Method combines these modules, iterating over them multiple times to improve prediction accuracy. The video mentions simplifying the inference process by excluding ensembling and template stacks. Key points include how recycling is handled by zero-initializing tensors in the first iteration and how new inputs are generated for each recycling step, adding variability to the process. Some troubleshooting tips are provided, including adjusting tensor datatypes and device settings. The series concludes with congratulations for completing the journey into AlphaFold and a promise of future developments, potentially with AlphaFold 3.

1. Mandon, K., AlphaFold Decoded Codebase. A comprehensive implementation and educational resource for understanding AlphaFold's architecture and functioning. Available from: https://gitlab.igem.org/2024/software-tools/aachen.

2. Jumper, J., et al., Highly accurate protein structure prediction with AlphaFold. Nature, 2021. 596(7873): p. 583-589. Available from: https://doi.org/10.1038/s41586-021-03819-2.

3. Ahdritz, G., et al., OpenFold: An Open-Source Implementation of AlphaFold. A valuable resource for code verification and utilizing pre-trained weights. Available from: https://github.com/aqlaboratory/openfold.

4. Senior, A.W., et al., AlphaFold Codebase. The official code repository that inspired and supported our implementation with code snippets and structural insights. Available from: https://github.com/google-deepmind/alphafold.

5. Mandon, K., AlphaFold Decoded YouTube Channel. Educational video series on AlphaFold, offering a step-by-step breakdown and explanation. Available from: http://www.youtube.com/@AlphaFoldDecoded.