FlockDB

FlockDB is a distributed graph database for storing adjancency lists, with goals of supporting: a high rate of add/update/remove operations potientially Complex set arithmetic queries paging through query result sets containing millions of entries ability to "archive" and later restore archived edges horizontal scaling including replication online data migration Non-goals include: multi-Hop queries (or graph-walking queries) automatic shard migrations FlockDB is much simpler than other graph databases such as neo4j because it tries to solve fewer problems. It scales horizontally and is designed for on-line, low-latency, high throughput environments such as web-sites. Twitter uses FlockDB to store social graphs (who follows whom, who blocks whom) and secondary indices. As of April 2010, the Twitter FlockDB Cluster stores 13+ billion edges and sustains peak traffic of 20k writes/second and 100k reads/second. It does what?If, for example, you`re storing a social graph (user A follows user B), and it`s not necessarily symmetrical (A can follow B without B following A), then FlockDB can store that relationship as an edge: node A points to node B. It stores this edge with a sort position, and in both directions, so that it can answer the question "Who follows A?" as well as "Whom is A following?" This is called a directed graph. (Technically, FlockDB stores the adjacency lists of a directed graph.) Each edge has a 64-bit source ID, a 64-bit destination ID, a state (normal, removed, archived), and a 32-bit position used for sorting. The edges are stored in both a forward and backward direction, meaning that an edge can be queried based on either the source or destination ID. For example, if node 134 points to node 90, and its sort position is 5, then there are two rows written into the backing store: forward: 134 -> 90 at position 5backward: 90