Abbreviated listing of src/txtai ann api database embeddings pipeline scoring vectors workflow

# Install txtaipip install txtai datasets

from datasets import load_datasetdataset = load_dataset("ag_news", split="train")

import numpy as npfrom txtai.vectors import VectorsFactorymodel = VectorsFactory.create({"path": "sentence-transformers/all-MiniLM-L6-v2"}, None)embeddings = []# List of all text elementstexts = dataset["text"]# Create embeddings buffer, vector model has 384 featuresembeddings = np.zeros(dtype=np.float32, shape=(len(texts), 384))# Vectorize text in batchesbatch, index, batchsize = [], 0, 128for text in texts:  batch.append(text)  if len(batch) == batchsize:    vectors = model.encode(batch)    embeddings[index : index + vectors.shape[0]] = vectors    index += vectors.shape[0]    batch = []# Last batchif batch:    vectors = model.encode(batch)    embeddings[index : index + vectors.shape[0]] = vectors# Normalize embeddingsembeddings /= np.linalg.norm(embeddings, axis=1)[:, np.newaxis]# Print shapeembeddings.shape

(120000, 384)

from txtai.ann import ANNFactory# Create Faiss index using normalized embeddingsann = ANNFactory.create({"backend": "faiss"})ann.index(embeddings)# Show totalann.count()

120000

query = model.encode(["best planets to explore for life"])query /= np.linalg.norm(query)for uid, score in ann.search(query, 3)[0]:  print(uid, texts[uid], score)

17752 Rocky Road: Planet hunting gets closer to Earth Astronomers have discovered the three lightest planets known outside the solar system, moving researchers closer to the goal of finding extrasolar planets that resemble Earth. 0.59904360771179216158 Earth #39;s  #39;big brothers #39; floating around stars Washington - A new class of planets has been found orbiting stars besides our sun, in a possible giant leap forward in the search for Earth-like planets that might harbour life. 0.568852901458740245029 Coming Soon: "Good" Jupiters Most of the extrasolar planets discovered to date are gas giants like Jupiter, but their orbits are either much closer to their parent stars or are highly eccentric. Planet hunters are on the verge of confirming the discovery of Jupiter-size planets with Jupiter-like orbits. Solar systems that contain these "good" Jupiters may harbor habitable Earth-like planets as well. 0.5606889724731445

from txtai.embeddings import Embeddingsembeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"})embeddings.index((x, text, None) for x, text in enumerate(texts))for uid, score in embeddings.search("best planets to explore for life"):  print(uid, texts[uid], score)

17752 Rocky Road: Planet hunting gets closer to Earth Astronomers have discovered the three lightest planets known outside the solar system, moving researchers closer to the goal of finding extrasolar planets that resemble Earth. 0.59904360771179216158 Earth #39;s  #39;big brothers #39; floating around stars Washington - A new class of planets has been found orbiting stars besides our sun, in a possible giant leap forward in the search for Earth-like planets that might harbour life. 0.56885296106338545029 Coming Soon: "Good" Jupiters Most of the extrasolar planets discovered to date are gas giants like Jupiter, but their orbits are either much closer to their parent stars or are highly eccentric. Planet hunters are on the verge of confirming the discovery of Jupiter-size planets with Jupiter-like orbits. Solar systems that contain these "good" Jupiters may harbor habitable Earth-like planets as well. 0.560688853263855

from txtai.database import DatabaseFactory# Load content into databasedatabase = DatabaseFactory.create({"content": True})database.insert((x, row, None) for x, row in enumerate(dataset))# Show totaldatabase.search("select count(*) from txtai")

[{'count(*)': 120000}]

database.search("select count(*), label from txtai group by label")

[{'count(*)': 30000, 'label': 0}, {'count(*)': 30000, 'label': 1}, {'count(*)': 30000, 'label': 2}, {'count(*)': 30000, 'label': 3}]

for row in database.search("select id, text from txtai where text like '%planets%' limit 3"):  print(row["id"], row["text"])

100 Comets, Asteroids and Planets around a Nearby Star (SPACE.com) SPACE.com - A nearby star thought to harbor comets and asteroids now appears to be home to planets, too. The presumed worlds are smaller than Jupiter and could be as tiny as Pluto, new observations suggest.102 Redesigning Rockets: NASA Space Propulsion Finds a New Home (SPACE.com) SPACE.com - While the exploration of the Moon and other planets in our solar system is nbsp;exciting, the first task for astronauts and robots alike is to actually nbsp;get to those destinations.272 Sharpest Image Ever Obtained of a Circumstellar Disk Reveals Signs of Young Planets MAUNA KEA, Hawaii -- The sharpest image ever taken of a dust disk around another star has revealed structures in the disk which are signs of unseen planets.     Dr...

from txtai.scoring import ScoringFactory# Build indexscoring = ScoringFactory.create({"method": "bm25", "terms": True, "content": True})scoring.index((x, text, None) for x, text in enumerate(texts))# Show totalscoring.count()

120000

for row in scoring.search("planets explore life earth", 3):  print(row["id"], row["text"], row["score"])

16327 3 Planets Are Found Close in Size to Earth, Making Scientists Think 'Life' A trio of newly discovered worlds are much smaller than any other planets previously discovered outside of the solar system. 17.76833244813070716158 Earth #39;s  #39;big brothers #39; floating around stars Washington - A new class of planets has been found orbiting stars besides our sun, in a possible giant leap forward in the search for Earth-like planets that might harbour life. 17.6594196817079316620 New Planets could advance search for Life Astronomers in Europe and the United States have found two new planets about 20 times the size of Earth beyond the solar system. The discovery might be a giant leap forward in  17.65941968170793

def search(query, limit=3):  # Get similar clauses, if any  similar = database.parse(query).get("similar")  return database.search(query, [scoring.search(args[0], limit * 10) for args in similar] if similar else None, limit)# Rebuild scoring - only need terms indexscoring = ScoringFactory.create({"method": "bm25", "terms": True})scoring.index((x, text, None) for x, text in enumerate(texts))for row in search("select id, text, score from txtai where similar('planets explore life earth') and label = 0"):  print(row["id"], row["text"], row["score"])

15363 NASA to Announce New Class of Planets Astronomers have discovered four new planets in a week's time, an exciting end-of-summer flurry that signals a sharper era in the hunt for new worlds.    While none of these new bodies would be mistaken as Earth's twin, some appear to be noticeably smaller and more solid - more like Earth and Mars - than the gargantuan, gaseous giants identified before... 12.58292325969713215900 Astronomers Spot Smallest Planets Yet American astronomers say they have discovered the two smallest planets yet orbiting nearby stars, trumping a small planet discovery by European scientists five days ago and capping the latest round in a frenzied hunt for other worlds like Earth.    All three of these smaller planets belong to a new class of "exoplanets" - those that orbit stars other than our sun, the scientists said in a briefing Tuesday... 12.56392823106715515879 Astronomers see two new planets US astronomers find the smallest worlds detected circling other stars and say it is a breakthrough in the search for life in space. 12.078383982352994