CLOUDLIGHTNING: A Framework for a Self-organising and Self-managing Heterogeneous Cloud

Abstract

As clouds increase in size and as machines of different types are added to the infrastructure in order to\nmaximize performance and power efficiency, heterogeneous clouds are being created. However, exploiting\ndifferent architectures poses significant challenges. To efficiently access heterogeneous resources and, at the\nsame time, to exploit these resources to reduce application development effort, to make optimisations easier\nand to simplify service deployment, requires a re-evaluation of our approach to service delivery. We\npropose a novel cloud management and delivery architecture based on the principles of self-organisation\nand self-management that shifts the deployment and optimisation effort from the consumer to the software\nstack running on the cloud infrastructure. Our goal is to address inefficient use of resources and\nconsequently to deliver savings to the cloud provider and consumer in terms of reduced power consumption\nand improved service delivery, with hyperscale systems particularly in mind. The framework is general but\nalso endeavours to enable cloud services for high performance computing. Infrastructure-as-a-Service\nprovision is the primary use case, however, we posit that genomics, oil and gas exploration, and ray tracing\nare three downstream use cases that will benefit from the proposed architecture.