Sources Contact Advanced Search Tutorials

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

January 16, 2023 11:21 pm GMT

Building the Ultimate Search for Rails - Episode 1

During summer 2021, I got lucky enough to cross Twitter paths with Peter Szinek, who introduced me to his team and got me hired at RCRDSHP, Obie Fernandezs latest Web 3.0 project involving Music NFTs. Being myself a pro musician and music producer, I was thrilled to finally be able to mix my two passions-turned-into-a-living. Surrounded by awesome developers, I learned and built tons of cool stuff, among which a reactive, super performant server-side-rendered search experience using StimulusReflex, ElasticSearch and close to no JavaScript. The feature is still live and pretty much unchanged.

The purpose of this series will be to first reimplement this super friendly UX with basic filters and sort options, along with StimulusReflex. Then, well see how ElasticSearch can allow more complex filters and search scenarios, while improving performance. In the last episode, well try and replace StimulusReflex with the new Custom Turbo Stream Actions and compare implementation/behaviour. If time allows, I might add a bonus episode to show how to deploy all this in production. Lets dig in.

What are we building?

Remember how back in the day, people used to collect art printed on actual paper ? A bit like NFTs, only physical. Weird, right? Well, lets picture an app that would allow users to buy and sell limited edition art prints. The prints would include photographs, movie posters, and illustrations of various formats. It should look like that:

Heres what the DB looks like:

Please note the tags column is of string array type. One might argue that the Listing table, in our case, could easily be skipped. But for the sake of keeping a real-world complexity scenario, lets say that wed like to keep the actual Prints separate from their listings (and since the app allows users to sell their prints, we might reasonably think that a print could be listed several times).

OK. Show me the gear

First things first: on the frontend, well use StimulusReflex (a.k.a SR) to build a super reactive and friendly search experience with very little code, and little to no JavaScript. For those unfamiliar:

StimulusReflex is a library that extends the capabilities of both Rails and Stimulus by intercepting user interactions and passing them to Rails over real-time websockets. The current page is quickly re-rendered and morphed to reflect the new application state.

Sounds a bit like Hotwire on paper, though youll see how their philosophy greatly differs in the last episode of this series. Well also use a sprinkle of CableReady, a close cousin of SR.

On the backend, we'll need a few tools. Apart from the classics (ActiveRecord scopes and the pg_search gem), youll see how the (yet officially unreleased but production-tested) all_futures gem, built by SR authors, will act as an ideal ephemeral object to temporarily store our filter params and host our search logic. Finally, well use pagy for pagination duties.

Philtre d'amour

(Please indulge this shitty French pun, the expression philtre d'amour meaning love potion but also sounds like beloved filter)

Lets start by creating some simple data. Well add a few artworks of different kind : photographs, illustrations, and posters. Each will have several tags from a given list and an author. For now, lets just generate one print per artwork, and a listing for each. Prints can be one of 3 available formats , while listings will be of varying price.
Now lets list our different features:

Search by name or author
Filter by minimum price
Filter by maximum price
Filter by category
Filter by print format
Filter by tags
Order by price
Order by date listed

Before I started building my feature, my former colleague and friend @marcoroth pointed me to leastbads Beast Mode, from which I took heavy inspiration to get going. Thats how I discovered his gem all_futures, which provides us with an ActiveRecord-like object that will persist to Redis. Lets see how things look like.

# app/controllers/listings_controller.rbclass ListingsController < ApplicationController  def index    @filter ||= ListingFilter.create    @listings = @filter.results  endend# app/models/listing_filter.rbclass ListingFilter < AllFutures::Base  # Filters  attribute :query, :string  attribute :min_price, :integer, default: 1  attribute :max_price, :integer, default: 1000  attribute :category, :string, array: true, default: []  attribute :tags, :string, array: true, default: []  attribute :format, :string, array: true, default: []  # Sorting  attribute :order, :string, default: "created_at"  attribute :direction, :string, default: "desc"  def results    # TODO: Build a query out of these attributes  endend

Notice how params are absent from the controller, and nothing gets passed to our ListingFilter object? And how come @filter could potentially be already defined? Youll see why in a bit, so lets first look at building the query.

In his approach, @leastbad simply created an ActiveRecord scope for each filter, then very cleverly and neatly, chained them to build his final filtered query, much like this:

# In app/models/listing_filter.rbdef results  Listing.for_sale    .price_between(min_price, max_price)    .from_categories(category)    .with_tags(tags)    .with_formats(format)    .search(query)    .order(order => direction)end

You might wonder: But what if filters are empty and arguments blank? The chains gonna break!. Well, have a look at the scopes declaration:

# app/models/listing.rbclass Listing < ApplicationRecord  belongs_to :print  scope :for_sale,        ->{ where(sold_at: nil) }  scope :price_between,   ->(min, max) { where(price: min..max) }  scope :with_formats,    ->(format_options) { joins(:print).where(prints: {format: format_options}) if format_options.present? }  scope :from_categories, ->(cat_options) { joins(:artwork).where(artworks: {category: cat_options}) if cat_options.present? }  scope :with_tags,       ->(options) { joins(:artwork).where("artworks.tags && ?", "{#{options.join(",")}}") if options.present? }  scope :search           ->(query) { # TODO } end

In ListingFilter, the crucial bit is to make sure that every attribute has a default value. The magic then occurs in the if statement at the end of the scopes expecting an argument: if the lambda returns nil, then it will essentially be ignored, and the collection returned as is. Such a nice trick. Time for some specs to ensure that things actually work:

RSpec.describe ListingFilter, type: :model do  let!(:photo) { Artwork.create(name: "Dogs", author: "Elliott Erwitt", year: 1962, tags: %w[Animals B&W USA], category: "photography") }  let!(:poster) { Artwork.create(name: "Fargo", author: "Matt Taylor", year: 2021, tags: %w[Cinema USA], category: "poster") }  let!(:photo_print) { photo.prints.create(format: "30x40", serial_number: 1) }  let!(:photo_print_2) { photo.prints.create(format: "18x24", serial_number: 200) }  let!(:poster_print) { poster.prints.create(format: "40x50", serial_number: 99) }  let!(:photo_listing) { photo_print.listings.create(price: 800) }  let!(:photo_listing_2) { photo_print_2.listings.create(price: 400) }  let!(:poster_listing) { poster_print.listings.create(price: 200) }  let!(:sold_listing) { poster_print.listings.create(price: 300, sold_at: 2.days.ago) }  describe "#results" do    it "doesn't return a sold listing" do      expect(ListingFilter.create.results).not_to include(sold_listing)    end    context "Filter options" do      it "Filters by price" do        filter = ListingFilter.create(min_price: 100, max_price: 300)        expect(filter.results).to match_array([poster_listing])        filter.update(max_price: 1000)        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])      end      it "Filters by format" do        filter = ListingFilter.create(format: ["40x50"])        expect(filter.results).to match_array([poster_listing])        filter.update(format: ["40x50", "30x40"])        expect(filter.results).to match_array([photo_listing, poster_listing])      end      it "Filters by category" do        filter = ListingFilter.create(category: ["photography"])        expect(filter.results).to match_array([photo_listing_2, photo_listing])        filter.update(category: ["photography", "poster"])        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])      end      it "Filters by tags" do        filter = ListingFilter.create(tags: ["Cinema"])        expect(filter.results).to match_array([poster_listing])        filter.update(tags: ["Cinema", "Animals"])        expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])      end      it "Filters by multiple attributes" do        filter = ListingFilter.create(tags: ["Cinema"], max_price: 300, category: ["poster"])        expect(filter.results).to match_array([poster_listing])      end    end  endend

All green. I hope youll appreciate how easy it is to test this. Lets quickly add pg_search to our Gemfile, then take care of the search scope:

# app/models/listing.rbclass Listing < ApplicationRecord  include PgSearch::Model  belongs_to :print  has_one :artwork, through: :print  scope :search, ->(query) { basic_search(query) if query.present? }  # Skipping the other scopes...  pg_search_scope :basic_search,    associated_against: {      artwork: [:name, :author]    },    using: {      tsearch: {prefix: true}    }  #...end

Since our listings dont carry much information, well have to jump a few tables to look where we need, namely the name and author columns of our Artwork model. Unfortunately, pg_search doesnt support associated queries further than 1 table away, thus the has_one... through relationship we needed to add. Lets add some tests for the search:

context "Search" do  it "renders all listings if no query is passed" do    filter = ListingFilter.create(query: "")    expect(filter.results).to match_array([photo_listing, photo_listing_2, poster_listing])  end  it "can search by artwork name or author" do    filter = ListingFilter.create(query: "Erwitt")    expect(filter.results).to match_array([photo_listing, photo_listing_2])    filter.update(query: "Fargo")    expect(filter.results).to match_array([poster_listing])  end  it "can both search and sort" do    filter = ListingFilter.create(query: "Erwitt", order_by: "price", direction: "asc")    expect(filter.results.to_a).to eq([photo_listing_2, photo_listing])    filter.update(query: "Erwitt", order_by: "price", direction: "desc")    expect(filter.results.to_a).to eq([photo_listing, photo_listing_2])  endend

We run the tests and of course, everything is gr Oh no. Looks like the last test is acting up:

Apparently, a known problem of pg_search is that it doesnt play well with eager loading, nor combinations of join and where queries. The recommended workaround (and my usual plan B when ActiveRecord queries start to get ugly) is to use a subquery:

# In app/models/listing_filter.rbdef results  filtered_listings_ids = Listing.for_sale    .price_between(min_price, max_price)    .from_categories(category)    .with_tags(tags)    .with_formats(format)    .pluck(:id)  Listing.where(id: filtered_listings_ids)    .search(query)    .order(order_by => direction)    .limit(200)end

Lets also add some last specs for the sort options and run all this.

context "Sort options" do  specify "Recent listings first (default behaviour)" do    filter = ListingFilter.create    expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])  end  specify "Most expensive first" do    filter = ListingFilter.create(order_by: "price", direction: "desc")    expect(filter.results.to_a).to eq([photo_listing, photo_listing_2, poster_listing])  end  specify "Least expensive first" do    filter = ListingFilter.create(order_by: "price", direction: "asc")    expect(filter.results.to_a).to eq([poster_listing, photo_listing_2, photo_listing])  endend

Everythings green Except for the search and filter option. The errors gone, but the test still fails; the ordering doesnt seem to work, despite all the sorting tests being green. After another lookup on pg_search known issues, it appears that order statements following the search scope dont work. Workarounds include using reorder instead, or moving the order clause up the chain. I opted for the first option, which make all tests pass. Let's move on.

Stairway to Heaven

Now that we know that our backend is working as it should, lets wire up our stuff. Im gonna skip on Stimulus Reflex setup and configuration and dive right in. You can easily follow the official setup or, if you use import-maps, follow @julianrubischs article on the topic. I also know that leastbad has been working on an automatic installer that detects your configuration and sets everything up for you if you care to try it before the next version of SR gets released.

Once youre done with that, lets begin with the sort first. Lets recap our sorting options and store them somewhere:

class ListingFilter < AllFutures::Base  SORTING_OPTIONS = [    {column: "created_at", direction: "desc", text: "Recently added"},    {column: "price", direction: "asc", text: "Price: Low to High"},    {column: "price", direction: "desc", text: "Price: High to Low"}  ]  #...  attribute :order_by, :string, default: "created_at"  attribute :direction, :string, default: "desc"  #...  # Memoizing the value to avoid re-computing at every call  def selected_sorting_option    @_selected_option ||= SORTING_OPTIONS.find {|option| order_by == option[:column] && direction == option[:direction] }  endend

Then in our Sort by dropdown, well have something like :

<div class="dropdown">  <button>    Sort by:<span><%= @filter.selected_sorting_option[:text] %></span>  </button>  <!-- Skipping lots of HTML -->  <% ListingFilter::SORTING_OPTIONS.each do |option| %>    <% if option == @filter.selected_sorting_option %>      <span class="font-semi-bold ..."><%= option[:text] %></span>    <% else %>      <a data-reflex="click->Listing#sort"        data-column="<%= option[:column] %>"        data-direction="<%= option[:direction] %>"        data-filter-id="<%= @filter.id %>"        href="#"      >        <%= option[:text] %>      </a>    <% end %>  <% end %></div>

Even if you're unfamiliar with StimulusReflex, it should still remind you of the way we invoke regular stimulus controllers. Only here, when our link gets clicked, it should trigger the sort action (a ruby method) from the Listing reflex (a ruby class). Lets code it:

# app/reflexes/listing_reflex.rbclass ListingReflex < ApplicationReflex  def sort    @filter = ListingFilter.find(element.dataset.filter_id)    @filter.order_by = element.dataset.column    @filter.direction = element.dataset.direction    @filter.save  endend

And sure enough, it works! So what's going on here? Well, clicking the link invokes our reflex, which gets executed right before our current controller action runs again. It allows us to execute any kind of server-side logic, as well as play with the DOM in various ways, but with ruby code. Then, the DOM gets morphed over the wire.

What we did in our specific case: since our filter object is being persisted in Redis, it has a public id, which we stored as a data-attribute, and later retrieved from our reflex action. Then, we fetched the object from memory and updated it with new attributes. This is why @filter will be already defined by the time we get to that point. By default, not specifying anything more in our action will cause SR to just re-render the whole page before running the controller action. We could be more specific here, and just choose to morph a few elements to save precious milliseconds. But for demo purposes well leave it as is.

Lets add a filter next. Well start with the first one, by minimum price.

<div class="text-sm text-gray-600 flex justify-between">  <label for="min-price">Minimum Price:</label>  <span><output id="minPrice">50</output> $</span></div><input type="range"  data-reflex="change->Listing#min_price"  data-filter-id="<%= @filter.id %>"  name="min-price"  min="50"  max="1000"  value="<%= @filter.min_price %>"  class="accent-indigo-600"  oninput="document.getElementById('minPrice').value = this.value">

I got lazy and didnt want to code an extra stimulus controller just to show the price value. But apart from that, we just need to add the new #min_price action:

# app/reflexes/listing_reflex.rbclass ListingReflex < ApplicationReflex  def sort    @filter = ListingFilter.find(element.dataset.filter_id)    @filter.order_by = element.dataset.column    @filter.direction = element.dataset.direction    @filter.save  end  def min_price    @filter = ListingFilter.find(element.dataset.filter_id)    @filter.min_price = element.dataset.value    @filter.save      endend

And here in action:

I think by now you get the picture. Lets just do the search and one of the checkbox filters.

In the view:

<!-- Search --><input type="search" value="<%= @filter.query %>" data-filter-id="<%= @filter.id %>" data-reflex="change->Listing#search"><!-- Format Filter --><% Print::FORMATS.each_with_index do |format, index| %>  <div class="flex items-center">    <input data-reflex="change->Listing#format" <%= "checked" if @filter.format.include? format %> data-filter_id="<%= @filter.id %>" value="<%= format %>" type="checkbox">  </div><% end %>

Our Reflex actions are starting to be pretty similar to each other, which calls for a refactor. You cant do any better than leastbads approach, especially if you start having more complicated logic going on (like custom morphs or pagination):

# app/reflexes/listing_reflex.rbclass ListingReflex < ApplicationReflex  def sort    update_listing_filter do |filter|      filter.order_by = element.dataset.column      filter.direction = element.dataset.direction    end  end  def min_price    update_listing_filter do |filter|      filter.min_price = element.value.to_i    end  end  def max_price    update_listing_filter do |filter|      filter.max_price = element.value.to_i    end  end  def format    update_listing_filter do |filter|      filter.format = element.value    end  end  def search    update_listing_filter do |filter|      filter.query = element.value    end  end  private  def update_listing_filter    @filter = ListingFilter.find(element.dataset.filter_id)    yield @filter    @filter.save    # Add custom morphs here or any logic before the controller action is run  endend

And so on with the other filters. We can now combine search, filters and sort options with no page refresh.

Ambrosia on the cake

Lets enhance the UX a bit. Right now theres no pagination. Straight after adding pagy, clicking any page link will navigate, causing the params to reset. Lets fix this by overriding pagys default template and wire links to our Reflex instead:

 <!-- views/listings/_pagy_nav.html.erb --><% link = pagy_link_proc(pagy) -%><%#                            -%><nav class="pagy_nav pagination space-x-4" role="navigation"><% if pagy.prev                -%>  <span class="page prev"><a class="text-indigo-400" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.prev || 1 %>">Previous</a></span><% else                        -%>  <span class="page prev text-gray-300">Previous</span><% end                         -%><% pagy.series.each do |item|  -%><%   if    item.is_a?(Integer) -%>  <span class="page"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= item %>"><%== item %></a></span><%   elsif item.is_a?(String)  -%>  <span class="page page-current font-bold"><%= item %></span><%   elsif item == :gap        -%>  <span class="page text-gray-400"><%== pagy_t('pagy.nav.gap') %></span><%   end                       -%><% end                         -%><% if pagy.next                -%>  <span class="page next"><a class="text-indigo-400 hover:text-indigo-600" href="#" data-reflex="click->Listing#paginate" data-filter-id="<%= filter.id %>" data-page="<%= pagy.next || pagy.last %>">Next</a></span><% else                        -%>  <span class="page next disabled">Next</span><% end                         -%><%#                            -%></nav>

# Add the paginate method to ListingReflexdef paginate  update_listing_filter do |filter|    filter.page = element.dataset.page  endend# And update the controllerdef index  @filter ||= ListingFilter.create  @pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])  # Can sometimes happen over navigation when collection gets changed in real timerescue Pagy::OverflowError  @pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])end

Another issue is that at the moment, updating filters and our sorting options dont update the URL params; refreshing the page clears everything, and were not able to save or share the result of our search to someone. Lets take care of that as well. What we want is for our URL to always reflect the current state of filters on one hand, then be able to load our filter params from the URL on the other hand.

First step is made easy by the mighty cable_ready library, namely its push_state operation. Not only is it almost magical, but it is ready to use in any Reflex. Have a look at all you can do with it. Here is what our main action needs to do what we want:

# reflexes/listing_reflex.rbdef update_listing_filter  @filter = ListingFilter.find(element.dataset.filter_id)  yield @filter  @filter.save  # Updating URL with serialized attributes from our filter  cable_ready.push_state(url: "#{request.path}?#{@filter.attributes.to_query}")end

Now if you change any filter, type any query, change page or switch sorting option, the URL will update itself, including every filter attribute. Our last step is to load these attributes from the params on initial page load:

class ListingsController < ApplicationController  include Pagy::Backend  def index    @filter ||= ListingFilter.create(filter_params)    @pagy, @listings = pagy(@filter.results, items: 12, page: @filter.page, size: [1,1,1,1])  rescue Pagy::OverflowError    @pagy, @listings = pagy(@filter.results, items: 12, page: 1, size: [1,1,1,1])  end  private  # Don't forget to update this list when adding filter options  def filter_params    params.permit(      :query,      :min_price,      :max_price,      :page,       :order_by,       :direction,      category: [],      tags: [],      format: []    )  endend

Its starting to get pretty nifty. One last issue UX-wise : since we can no longer refresh to clear it all, we lack a Clear All button. Just add a link, then wire it to a Reflex action such as:

def clear  ListingFilter.find(element.dataset.filter_id).destroy  @filter = ListingFilter.create  cable_ready.push_state(url: request.path)end

And here you are, as close as ever to eternal bliss in search paradise. You can have a look at the live app here.

Behold the afterlife

Lets recap what we learned. Thanks to StimulusReflex , we learned how to build a super reactive search and filter interface with clean and extendable code, great performance, and almost no JavaScript. We saw how cable_ready could provide some sprinkle of magic behaviour on top of StimulusReflex. We were able to cleanly and temporarily persist, then update our search data thanks to all_futures. We also learned how to chain conditional scopes in a safe manner.

Unfortunately, good things rarely last forever. In our next episode, well see how new requirements and a bigger set of records will party poop our not-so-eternal dream. You'll get to see how ElasticSearch can save the day and allow us to build the ultimate search engine.

Thanks for reading folks, and see you on the other side!

Resources

Original Link: https://dev.to/lso/the-ultimate-search-for-rails-episode-1-1mi

Share this article:

View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To