Kora now powers 30,000+ Confluent Cloud fully managed clusters for thousands of customers, keeping their critical workloads running seamlessly, while reducing their time and money spent on operations and infrastructure management. Optimize the cost structure of large-scale usage Improve performance by deeply optimizing for the cloud environment and unique workloads of a streaming system in the cloudĬapture real-time usage to automatically optimize data placement, fault detection and recovery, and other aspects of operations Intelligently manage data locality between memory, SSDs, and object storage to maximize throughput while minimizing cost and latency Run across more than 85 regions in three cloudsīe operated at scale by a small on-call team (we estimate we are about 1000x more efficient at operations than the average Kafka team)ĭisaggregate individual components within the network, compute, metadata, and storage layer It is 100% compatible with all currently supported versions of the Kafka protocol, but is designed from the ground up to be a true managed service.īe multi-tenant first, supporting thousands of tenants with strong isolation We realized that to make our vision a reality we had to build something designed for these constraints. ![]() But there are also some things that are easier: it doesn’t need to be downloaded and installed so it can have multiple services each with a fair amount of complexity, it only runs on a tightly controlled hardware and software stack, it doesn’t need to be configurable for different environments, and it never needs to be operated by anyone but the team who builds it. Some of the challenges are much bigger: it has to serve thousands of customers, be built from the ground up to be multitenant, be largely driven by data-driven software-based operations rather than human operators, provide strong isolation and security across customers with unpredictable and even adversarial workloads, and be something a team of hundreds can innovate in rapidly. Apache Kafka is designed to be as easy as possible to set up and run with a variety of DevOps tools, provide a single cluster with high performance in any environment, be upgraded every year or so, serve users within a single company that works together to plan capacity, and be operable by generalist operations and engineering personnel.Ī cloud system has very different constraints. Why are these systems so different? Well, any system design ends up being defined by the constraints it has to satisfy, and the challenges facing a cloud data system are quite different from a self-managed open source download. These systems are completely different in how they are architected, how they operate, and the experience they deliver to customers. S3 isn’t anything like running a dedicated NFS server per customer, and Snowflake isn’t just a bunch of Teradata warehouses on EC2. The systems that have succeeded in fulfilling the promise of the cloud are very different beasts. We quickly realized that to build the service we imagined it wouldn’t be enough to just put Kafka on servers in AWS. It was the best Kafka offering of its day, but still far from what we envisioned. ![]() But despite the work we put into it, our early Kafka offering was far from that-it was basically just open source Kafka on a Kubernetes-based control plane with simplistic billing, observability, and operational controls. When we launched Confluent Cloud in 2017, we had a grand vision for what it would mean to offer Kafka in the cloud. In the rest of this blog post, I’ll describe why we built it, some of the advantages it gives, and a bit about how it works under the covers. It doesn’t in any way displace open source Kafka for those wanting a system they can manage themselves-and we continue to contribute heavily to Apache Kafka-but it is what allows us to offer a true cloud service at phenomenal scale. Kora isn’t something you can download, or even something you could run outside our control plane and the rest of our cloud, but understanding how it works gives an idea of some of what goes into our service and some of the advantages it conveys. Kora is a cloud data service that serves up the Kafka protocol for our thousands of customers and their tens of thousands of clusters. Powering this is a next-generation engine, Kora. Today I’d like to give a tour of the internals of Confluent’s Apache Kafka® service.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |