Intro

My latest side Rails project is Search Engine.

dato.rss is a simple, clean and fast RSS search engine with a RESTful API.

the project is divided into

Search Engine: Quickly search through the millions of available RSS feeds.

RESTful API: Turns feed data into an awesome API. The API simplifies how you handle RSS, Atom, or JSON feeds. You can add and keep track of your favourite feed data with a simple, fast and clean REST API. All entries are enriched by Machine Learning and Semantic engines.

Search

Search is just implemented with Full Text Search Postgres feature.

I used the pg_search Gem, which can be used in two ways:

Multi Search: Search across multiple models and return a single array of results. Imagine having three models: Product, Brand, and Review. Using Multi Search we could search across all of them at the same time, seeing a single set of search results. This would be perfect for adding federated search functionality to your app.

Search Scope: Search within a single model, but with greater flexibility.

for the implementation I found very useful the article https://pganalyze.com/blog/full-text-search-ruby-rails-postgres

    execute <<-SQL      ALTER TABLE entries      ADD COLUMN searchable tsvector GENERATED ALWAYS AS (        setweight(to_tsvector('simple', coalesce(title, '')), 'A') ||        setweight(to_tsvector('simple', coalesce(body,'')), 'B') ||        setweight(to_tsvector('simple', coalesce(url,'')), 'C')      ) STORED;    SQL

Search View

Feed Rank

Feed Ranking is provided by openrank a free root domain authority metric based on the common search pagerank dataset. The value is normilized by

((Math.log10(domain_rank) / Math.log10(100)) * 100).round

Machine Learning

Machine Learning is provided by dandelion API Semantic Text Analytics as a service, from text to actionable data. Extract meaning from unstructured text and put it in context with a simple API.

RESTful API

All API documentation is in the Wiki section. Feel free to make it better, of course.

https://github.com/davidesantangelo/dato.rss/wiki

To use some features such as adding a new feed you need a token with write permission. Currently only I can enable it. In case contact me

Github

curl 'https://datorss.com/api/searches?q=news' | json_pp{  "data": [    {      "id": "86b0f829-e300-4eef-82e1-82f34d03aff6"      "type": "entry"      "attributes": {        "title": "\"Pandemic, Infodemic\": 2 Cartoon Characters Battling Fake News In Assam"        "url": "https://www.ndtv.com/india-news/coronavirus-pandemic-infodemic-2-cartoon-characters-battling-fake-news-in-assam-2222333",        "published_at": 1588448805,        "body": "An English daily in Assam's