An Interest In:
Web News this Week
- April 28, 2024
- April 27, 2024
- April 26, 2024
- April 25, 2024
- April 24, 2024
- April 23, 2024
- April 22, 2024
Twitter Recommendation Algorithm is now open sourced
This requires a recommendation algorithm to distill the roughly 500 million Tweets posted daily down to a handful of top Tweets that ultimately show up on your devices For You timeline.
The pipeline above runs approximately 5 billion times per day and completes in under 1.5 seconds on average. A single pipeline execution requires 220 seconds of CPU time, nearly 150x the latency you perceive on the app.
Along side with OpenAI, I personally think this is one of an important moment in the computing community as no one would ever guess a global-scale algorithm such as Twitter's Recommendation becomes open-sourced. Based on their engineer blog post, it is not out of reach to say the code base literally costs hundred thousands if not millions a day to run. How do you feel about this moment?
Twitter's engineer blog
GitHub repository
twitter / the-algorithm
Source code for Twitter's Recommendation Algorithm
Twitter Recommendation Algorithm
The Twitter Recommendation Algorithm is a set of services and jobs that are responsible for constructing and serving theHome Timeline. For an introduction to how the algorithm works, please refer to our engineering blog. Thediagram below illustrates how major services and jobs interconnect.
These are the main components of the Recommendation Algorithm included in this repository:
Type | Component | Description |
---|---|---|
Feature | SimClusters | Community detection and sparse embeddings into those communities. |
TwHIN | Dense knowledge graph embeddings for Users and Tweets. | |
trust-and-safety-models | Models for detecting NSFW or abusive content. | |
real-graph | Model to predict likelihood of a Twitter User interacting with another User. | |
tweepcred | Page-Rank algorithm for calculating Twitter User reputation. | |
recos-injector | Streaming event processor for building input streams for GraphJet based services. | |
graph-feature-service | Serves graph features for a directed pair of Users (e.g. how many of User A's following liked Tweets from User B). | |
Candidate Source | search-index |
Original Link: https://dev.to/hunghvu/twitter-recommendation-algorithm-is-now-open-sourced-3p4m
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To