This overview is the first of a long series of posts about Raven replication.
Replication is the process of copying all data created or changed in one database to another database. Replication has two primary uses: It provides redundancy of data across physical servers, typically to achieve higher availability. And it enables distribution of requests across multiple physical servers, typically to achieve greater scale.
At a very high level, Raven replication is:
Database-scoped: A single Raven server can host multiple Raven databases. Replication is configured at the database level, not at the server level. Replication can be configured between two databases on the same server, or between two databases on different servers. (In practice, you do the latter, since that’s the only way to get the redundancy and scalability benefits.)
One-way: A replication relationship is always from a single source database to a single destination database. Master-master replication — where two databases stay in sync by replicating to each other — is composed from two completely independent one-way relationships.
Push-based: All replication requests are sent from the source database to the destination database.*
Asynchronous: As with indexing, replication occurs on a background thread, after (and outside) any transaction that was used to save a document in the source database.
Secure: The source database must authenticate with the destination database just like any other client, with Windows authentication or an API key (over https).
In a previous post I discussed how Raven uses Etags, including tracking the state of a replication relationship. Recapping from that post, here is a simple replication conversation between a source and destination database (over HTTP):
Source: [GET] What document Etag did you last get from me? Destination: 4 Source: [Sees that its most recent document Etag is 9.] Source: [POST] Here is a batch of documents (Etags 5 through 9). Destination: [Saves each document in the batch.] Destination: Ok, I have them. Source: [GET] What document Etag did you last get from me? Destination: 9 Source: [Waits until next execution of replication task.] ...
As you can see, each HTTP request is initiated from the source. The destination database records the last source document Etag it successfully received. Remember that this Etag is owned by the source database, and is unrelated to the destination’s own document Etags. The destination simply saves the source’s Etag, and returns it when asked.
You can also see that batching is involved. It would be inefficient to create a separate HTTP request for each document that needs to be replicated. The source includes up to 1024 documents or 25 MB in each POST (whichever limit is hit first). If more than 1024 documents or 25 MB need to be replicated, the source sends batches until it is caught up. [Updated batch size from 100 to 1024 to reflect recent code change. Added size limit.]
There is a single replication task running in the source database, which is executed each time a document changes. On each execution, it performs replication for each destination on a separate thread, so replication to multiple destinations happens in parallel. Document changes occurring after the replication task is already executing will be picked up in the next execution.
As long as there are unreplicated changes in the source for any destination, the replication task will keep executing again immediately after completing.
Once caught up, the source will check with each destination periodically, even when no document changes have occurred. The periodic check supports the case where the destination database was deleted and recreated — replication will begin again from scratch to that destination. (Since it is the destination’s job to track the state of the relationship, if you clear all data in the destination, replication will start over as if it had never taken place.) Periodic checks also provide early warning of a failure to contact the destination, even if there is nothing to replicate at the moment.
If the most recent replication run did have changes to replicate, then the periodic check is done 30 seconds later. However, if the most recent run found nothing to replicate, then checks are done every 5 minutes.
Attachments replicate in exactly the same way as documents. I have omitted them from the process and description above for simplicity. In reality, when the destination responds to the initial “what have you got?” request, it includes two Etags: one for attachments and one for documents. In each execution of the replication task, documents are replicated first, then attachments.
* There is one small exception to the rule that all replication communication is initiated from the source. When a Raven database is initialized (that is, when the server is starting), it notifies each of the other databases that have previously replicated inbound to it. This is intended to alert those sources that the destination is alive again after a restart or failure, so that they begin retrying replication immediately. I’ll cover this more in later posts on failure scenarios.