Can Appwrite handle 25 Million documents?

With the release of Appwrite 0.12 come many new features, most notable being a completely overhauled database service.

It's Here! Announcing the Release of Appwrite 0.12!

Eldad A. Fux for Appwrite Jan 5 6 min read

#javascript #webdev #programming #news

The database didn't scale well with many documents in previous Appwrite versions. This was noticeable in collections containing upwards of ten thousand documents.

Since Appwrite uses the same database for internal resources like Users and Files, this limitation impacted every other service indirectly. The more the data, the slower the requests.

With 0.12, the database API introduces Indexes, which allow you to improve queries for specific documents by a lot. Complimentary, we also added a new and more powerful way of filtering with different query operators.

These improvements along with a complete rewrite of the database service enhanced Appwrite in a way that we were able to effortlessly query a collection of 25 Million documents with excellent response times.

Benchmark

k6 was used with 500 Virtual Users (VUs) over 30 seconds, with each request performing an EQUALS query against an attribute that is indexed with a Key index.

import http from 'k6/http'import { Rate } from 'k6/metrics'const config = {    headers: {        'X-Appwrite-Key': '[API_KEY]',        'X-Appwrite-Project': '[PROJECT_ID]',        'Content-Type': 'application/json'    }};const myFailRate = new Rate('failed requests');export let options = {    vus: 500,    discardResponseBodies: true,    duration: '30s',    thresholds: {        'failed requests': ['rate<0.05'],    },};export default function () {    const res = http.get(        `http://[APPWRITE_URL]/v1/database/collections/[COLLECTION_ID]/documents?queries[]=number.equal(${Math.floor(Math.random() * 25000000 + 1)})`,        config    );    myFailRate.add(res.status !== 200);}

Collection

Here is a representation of what the collection looks like used for the benchmark, the $id is automatically generated, and number is a unique random value up to 25 million.

Request

Requests were performed against the list documents endpoint with an EQUALS query against the number attribute and a random number from 1 - 25,000,000. On top of the query itself, each request goes through authentication, authorization, and permissions validation.

/v1/database/[COLLECTION]/documents?queries[]=number.equal([RANDOM_NUMBER])

Results

The results speak for themselves. With 40k successful requests and 1,334 per second, we reached numbers that were very difficult to achieve with previous versions.

Not only is the new Database better at scale with simple Queries - but it also improves overall performance with more complex queries.

The Server were Appwrite was hosted had 8-Cores with 16GB memory and it's CPU usage during the benchmark peaked around 80%.

k6 was run on a Mac Mini M1 with 16 GB memory.

This performance boost is part of a series of steps we take to ensure developers can take full advantage of their Appwrite servers. We plan to share more data and insights from both the development process and Appwrite benchmarks with upcoming new versions.

Learn more

You can use following resources to learn more and get help:

• Appwrite
• Github
• Appwrite Docs
• Discord Community