In this section, we present an overview of state-of-the-art methods for replica consistency in cloud environments. This overview includes those methods that we considered to be among the most representative in the literature. Based on the similarities of their core ideas, we classified these methods into three distinct categories: (1) fixed consistency methods; (2) configurable consistency methods; and (3) consistency monitoring methods. In what follows, we first describe the generic characteristics of each category and then present an overview of its specific methods.
4.1 Fixed consistency methods
This category includes those methods that provide predefined, fixed consistency guarantees in cloud storage systems. Representative methods in this category are of two types: Event Sequencing-based Consistency and Clock-based Strict Consistency. They are described next.
4.1.1 Event sequencing-based consistency
Event sequencing-based consistency methods aim at hiding replication complexity based on the fact that transaction serializability is costly and often unnecessary in web applications [11]. Thus, they provide a simple, but in many situations, effective consistency guarantee solution.
The most representative system that implements this type of method is PNUTS [25], a massively parallel and geographically distributed DBMS developed by Yahoo!. Since Yahoo!’s web applications must provide a high degree of availability for their users and must be able to read data in the presence of failures, PNUTS not only provides these features but it also supports a high degree of fault tolerance, including network partitions. PNUTS architecture is divided into regions, which contain a complete copy of each table. Thus, multiple regions containing replicated data provides additional reliability. Furthermore, PNUTS stores structured metadata in directories, which implies that users can leverage on PNUTS scalability and low latency to ensure high performance for metadata operations, such as file creation, deletion and renaming. In short, PNUTS properly manages metadata without sacrificing scalability.
PNUTS developers observed that web applications typically manipulate one record at a time, whereas different records may be located in different geographic localities. Hence, an event sequencing-based consistency method establishes that all replicas of a given record receive all updates applied to that record in the same order. This strategy is implemented by designating one of the replicas as the master for each record, so that this master receives all writes sent to that record by the other replicas. If a record has the majority of its writes sent to a particular replica, this replica becomes the master for that record.
4.1.2 Clock-based strict consistency
Clock-based strict consistency methods are characterized by the use of clock-based mechanisms to control timestamps to enforce strict consistency [26, 29]. They offer the guarantee that arbitrary objects in the data store are accessed atomically and isolated from concurrent accesses. The approach behind this consistency method is based on the ability of a system to provide a timestamp log to track the order in which operations occur. According to Bravo et al. [18], this type of technique is implemented by the data storage systems themselves and might impact the consistency guarantees that they provide.
Spanner [26] and Clock-SI [29] are representative systems that implement this type of consistency method. Spanner is a scalable, globally-distributed NewSQL database service designed, built and deployed by Google. Spanner combines and extends ideas from two research communities: the database community and the systems community. From this last one, scalability and fault tolerance are the most representative features provided by Spanner. Since replication is used for global availability and geographic locality, applications can use Spanner for high availability, even in the face of wide-area natural disasters. Spanner allows different applications’ data to be partitioned across different sets of servers in the same datacenter. For this reason, partition tolerance is an important requirement in Spanner. In addition, Spanner provides constraints for controlling read and write latency. Spanner also assigns globally-meaningful commit timestamps that reflect the serialization order of the transactions, which may be distributed. Moreover, Spanner enforces that if a transaction T2 begins after the commit of a transaction T1, then the commit timestamp of T2 must be greater than the commit timestamp of T1.
On the other hand, Clock-SI provides a fully distributed protocol for partitioned data stores that supports availability and scalability, bringing performance benefits. It also avoids a “single point of failure” and, therefore, a potential performance bottleneck, thus improving transaction latency and throughput. Clock-SI implements the so called snapshot isolation replication, which is a consistency criterion for partitioned data storage. In this strategy, read-only operations read from a consistent snapshot and other operations perform a commit if no objects written by these transactions were concurrently written. The local physical clock is used by each transaction to identify its read timestamp.
Another example of a system that implements snapshot isolation replication is Vela [55], which is a system for running off-the-shelf relational databases on the cloud. Vela provides a primary master and two secondary replicas, which are synchronously replicated, thereby offering fault tolerance. In addition, Vela relies on hardware virtualization to improve performance by reducing complexity with a minimal cost in resources. Instead of using data replication for durability, Vela uses this technique for improving latency, decoupling the update and the read-only workloads. Moreover, Vela provides elasticity, by monitoring the CPU idle time percentage, and scalability, by adding new replicas according to its workload.
4.2 Configurable consistency methods
The methods included in this category implement mechanisms that provide configurable levels of consistency, such as self-adaptive or flexible consistency guarantees, which allow the selection or specification of the desired consistency level at any given point. These methods are of two types, Automated and Self-adaptive Consistency and Flexible Consistency, and are described next.
4.2.1 Automated and self-adaptive consistency
Automated and self-adaptive consistency methods aim at dynamically enforcing multiple consistency degrees over distinct data objects. Three main approaches have been adopted to implement them.
Stale reads estimation. This approach is based on an estimation of the rate of read operations that return instances of data objects that have already been updated to newer values - the stale reads [47]. Once the estimation model is computed, it is possible to identify the key parameter that affects the stale reads and then to scale up/down the number of replicas. This approach is mainly implemented by Harmony [23], which is a cloud storage system that automatically identifies the key parameters affecting the stale reads, such as system states and application requirements. Harmony adds minimal latency while reducing the stale data reads by almost 80%. Its goal is to gradual and dynamically tune the consistency level at run time according to the applications’ consistency requirements, in order to provide adequate tradeoffs between consistency and both performance and availability. An intelligent estimation model of stale reads is the key aspect of Harmony. Its mechanism of elastically scaling up/down the number of replicas maintains a minimal tolerable fraction of stale reads, which results on meeting the required level of consistency while achieving good performance.
Divergence bounds enforcement. This approach allows the evaluation and enforcement of divergence bounds over data objects (table/row/column), thus providing consistency levels that can be automatically adjusted based on statistical information [53]. The evaluation takes place on a divergence vector every time an update request is received, although it is necessary to identify the affected data objects. If any limit is exceeded, all updates since the last replication are placed in a FIFO-like queue to be propagated and executed on the other replicas. The most representative system that implements this approach is VFC3 (Versatile Framework for Consistency in Cloud Computing) [32], which adopts a consistency model for replicated data across datacenters. The VFC3 model considers the different data semantics and automatically adjusts the consistency levels based on statistical information. Its main goal is to offer control over consistency to provide high-availability without compromising performance. Furthermore, VFC3 targets cloud tabular data stores, offering rationalization of resources and improvement of Quality-of-Service (QoS), thereby reducing latency.
Dynamic allocation. This approach dynamically selects to which server (or even a set of servers) each read of a data item must be directed, so that the best service is delivered given the current configuration and system conditions. Hence, this approach is adaptable to distinct configurations of replicas and users, as well as to changing conditions, such as variations on the network performance or server load. Another important aspect of this approach is the fact that it allows application developers to provide a Service Level Agreement (SLA) that specifies the applications’ consistency/latency desires in a declarative manner. Pileus [57] is a key-value storage system that implements this approach by providing a diversity of consistency guarantee options for globally distributed and replicated data environments. In fact, Pileus allows several systems or clients of a system to achieve different consistency degrees, even when they share the same data. Moreover, Pileus supports availability and performance by limiting the set of suitable servers, whereby strong reads must be directed to the primary site and eventual reads can be answered by any replica. A large table can be split into one or more smaller tables in order to achieve scalability. In Pileus, users perform operations to access data that is partitioned and replicated among distinct servers, thus it must support partition tolerance and reliability.
4.2.2 Flexible consistency
Flexible consistency methods cover distinct approaches that adapt to predefined consistency models in a flexible way. There are four main approaches that implement such methods.
Invariants-based. This approach strengthens eventual consistency, thus allowing the applications to specify consistency rules, or invariants, that must be maintained by the system. Once those invariants are defined, it is possible to identify the operations that are potentially unsafe under concurrent execution, thus allowing one to select either a violation-avoidance or an invariant-repair technique. Indigo [12] is a middleware system that implements this approach. It supports an alternative consistency method built on top of a geo-replicated and partitioned key-value data store. In addition, Indigo guarantees strong application invariants, while providing low latency to an eventually-consistent system. Indigo builds on the fault tolerance of the underlying storage system, thereby the failure of a machine inside a datacenter does not lead to any data loss.
Linearizable/eventual. This approach supports updates with a choice of linearizable and eventual consistency. GEO [15] is an open-source geo-distributed actorFootnote 2 system that implements this approach. It supports both replicated and single-instance coherence protocols. Replication can provide fast, always-available reads and updates. GEO also improves performance by caching actor states in one or more datacenters. Furthermore, the geo-distributed actor system can reduce access latencies by exploiting locality, since the caching policy for each actor can be declared as single-instance or multi-instance. Caching multiple instances can reduce the access latency for actors without locality. In GEO, an actor may be declared as volatile or persistent. In the first case, the latest version resides in memory and may be lost when servers fail, whereas in the second case the latest version resides in the storage layer. In case a user requests linearizability, GEO guarantees true linearizability in real time, between call and return.
Requirements-based. This approach supports the applications’ consistency requirements. The discussion that follows adopts the notion of service (requested by an application) as a generalization of the read and write operations used thus far. The trade-off between consistency and scalability requirements is handled by introducing the notion of consistency regions and service-delivery-oriented consistency policies. A consistency region is defined as a logical unit that represents the application state-level requirements for consistency and scalability. This concept is used to define consistency boundaries that separate each group of services that need to be ordered. Hence, services that need to be kept consistent must be associated to a certain region. The definition of what region a service belongs to and which services that can be concurrently delivered is set by the system administrators. Scalable Service Oriented Replication (SSOR) [22] is a middleware that implements this approach. SSOR presents a Region-based Election Protocol (REP) that provides a mechanism to balance the workload amongst sequencers, thereby efficiently improving elasticity in the cloud. Replication is used by SSOR to provide end-to-end fault tolerance between end clients and cloud services. In order to ensure reliability, SSOR implements solutions that tolerate sequencers and nonsequencers crashes, and introduces the concept of region distribution synchrony for handling simultaneous node crashing. A better performance is also achieved by reducing the load on the sequencer, extending the Multi-fixed Sequencers Protocol (MSP).
SSOR covers three distinct types of consistency region: (1) Conflict Region (CR), which is a region composed by services that have conflicting requirements for consistency regardless of the session; (2) Sessional Conflict Region (SCR), which is a region that includes services of a particular session with conflicting consistency requirements; and (3) Non-Conflict Region (NCR), which is a region that does not impose any consistency constraints or requirements.
Adaptable. This approach handles unpredictable workloads by allowing the system to be tuned for capacity in an elastic and flexible way. Due to this characteristic, it allows applications to perform eventually or strongly consistent reads as needed. Amazon DynamoDBFootnote 3 [2] is a highly reliable and cost-effective NoSQL database service that implements this approach. DynamoDB provides high availability and high throughput at very low latency. It has been designed to be scalable and to achieve high performance even at high scale. It was built based on the experience with its predecessor Dynamo [28]. DynamoDB adopts eventual consistency as its default model, which does not guarantee that an eventually consistent read will always reflect the result of a recently completed write. On the other hand, when adopting a stronger consistency model, it returns a result that reflects all writes that have received a successful response prior to that read.
4.3 Consistency monitoring methods
Alternatively, instead of directly handling data consistency issues, some methods focus on providing mechanisms that allow data owners to detect the occurrence of consistency violations in the cloud storage. This means that clients might audit their own data and make decisions based on how the Cloud Service Provider (CSP) stores and manages their replicas according to the consistency level that has been agreed upon in the service level contract. The methods in this category are of two types, Consistency Verification and Consistency Auditing, and are described next.
4.3.1 Consistency verification
Consistency verification methods are based on two approaches, namely protocol-based and contract-based.
Protocol-based. This approach is based on a protocol that enables a group of mutually trusting clients to detect consistency violations on a cloud storage. It is adopted by VICOS (Verification of Integrity and Consistency for Cloud Object Storage) [17]. VICOS supports the concept of fork-linearizability, which captures the strongest achievable notion of consistency in multi-client models. The method may guarantee this notion by registering the causal evolution of the user’s views into their interaction with the server. When the server creates only a single discrepancy between the views of two clients, it is ensured that these clients will never observe each other’s operations afterwards. That is, if these users later communicate and the server lies to them, the violation will be immediately discovered. Thus, users can verify a large number of past transactions by performing a single check.
Contract-based. This approach provides a verification scheme that allows data owners to ensure whether the CSP complies with the SLA for storing data in multiple replicas. It is implemented by DMR-PDP (Dynamic Multi-Replica Provable Data Possession) [51]. The context addressed by this scheme is that whenever data owners ask the CSP to replicate data at different servers, they are charged for this. Hence, data owners need to be strongly persuaded that the CSP stores all data copies that are agreed upon in the service level contract, as well as that all remotely stored copies correctly execute the updates requested by the users. This approach deals with such problems by preventing the CSP from cheating the data storage, for instance, by maintaining fewer copies than paid for. Such scheme is based on a technique called Provable Data Possession [9], which is used to audit and validate the integrity and consistency of data stored on remote servers.
4.3.2 Consistency auditing
This type of method is based on an architecture that consists of a large data cloud maintained by a CSP and multiple small audit clouds composed of a group of users that cooperate on a specific job (e.g., revising a document or writing a program). The required level of consistency that should be provided by the data cloud is stipulated by an SLA involving the audit cloud and the data cloud. Once the SLA is defined, the audit cloud can verify whether the data cloud violates it, thus quantifying, in monetary terms or otherwise, the severity of the violation.
Consistency as a Service (CaaS) [46, 49] implements this method. It relies on a two-level auditing structure, namely: local auditing and global auditing. Local auditing allows each user to independently perform local tracing operations, focusing on monotonic read and read-your-write consistencies. Global auditing, on the other hand, requires that an auditor be periodically elected from the audit cloud to perform global tracing operations, focusing on causal consistency. This method is supported by constructing a directed graph of operations, called the precedence graph. If the graph is acyclic, the required level of consistency is preserved [46].