Deployment Models
Different ways you can deploy a Sycomp Storage cluster
Last updated
© 2024 Sycomp A Technology Company, Inc. All rights reserved.
Different ways you can deploy a Sycomp Storage cluster
Last updated
When deploying Sycomp Storage you can choose the amount of storage, the number of VMs running the IBM Storage Scale native client, the machine types to use and other options. Optionally you can use NFS or install the IBM Storage Scale client on your application nodes to give them access to the data.
Figure 1 is an example of a Sycomp Storage deployment in Google Cloud. The Sycomp tools deploy the IBM Storage Scale cluster and a set of VMs running IBM Storage Scale native client that can be used to run your HPC or big data application. In this example Google balanced persistent storage was used for reliability, you can choose what type of Persistent Disk you want to use.
The VMs running IBM Storage Scale Native Client and the VMs using NFS can concurrently access the data stored in the Sycomp Storage cluster. All resources are deployed within a single project.
You can reduce the number of Network Shared Disk (NSD) server VMs needed to reach a performance target by using Google Local SSD.
Local SSD is ephemeral storage that offers a high number of input and output operations per second (IOPS) and low latency. To prevent data loss, Sycomp Storage leverages Storage Scale replication for Local SSD storage pools. With replication enabled the storage pool can survive the failure of a storage VM or Local SSD. In the event of a failure Storage Scale automatically creates new replicas to prevent against future failures.
Whether you want to deploy multiple machine types, add a set of application nodes to handle increased year end processing, or quickly spin up a DR application cluster an application cluster is a good way to manage your compute nodes.
Figure: Example of an application cluster, once deployed this cluster is attached to a Sycomp Storage cluster containing a file system.
A Storage Scale application cluster accesses an existing file system using Storage Scale multi-cluster. Sycomp Storage tools make it easy to maintain application clusters.
Your deployment can be enhanced by enabling data encryption with an HA pair of VMs running IBM Security Guardium Key Lifecycle Manager (GKLM) to manage and protect your encryption keys.
Sycomp Storage automates the deployment of an HA pair of GKLM server VMs.
You can add storage, NFS servers and clients to an existing Sycomp Storage cluster. Adding storage capacity is achieved by adding NSD servers. When you add NSD servers to the cluster each new servers gets the same disk configuration as the existing NSD servers. Storage is added in this manner to maintain balance across NSD servers. You can add NFS servers to increase capacity. Adding clients in groups allows you to customize the names and machine types of each set of clients giving you more control over your deployment.
Sycomp Storage ensures cluster expansion maintains IBM Storage Scale best practices when storage or nodes are added to the cluster by automatically deploying the correct version of software, placing disks in the correct pools and evenly distributing disks across NSD servers for optimal performance.
Sycomp Storage allows you to create a copy of a remote data set that exists in Google Cloud Storage or an NFS share in a high-performance POSIX file system. The caching tool has an extensive set of features.
Data on demand - File metadata and data is copied into the Sycomp Storage file system when a file is accessed. A directory listing in the Sycomp Storage file system fetches the file metadata, and a file open initiates a copy of the file from the remote NFS or Object store.
Data Prefetch - To keep from wasting GPU time you can ask Sycomp Storage to prefetch a set of data before your compute job starts. You can prefetch a directory tree or provide a custom list of files.
Read-only - You can create a read-only copy of the source data or a read-only copy that allows local modifications but never pushes the changes back to the source.
Writable - You can create a data copy that pushes changes made in the Sycomp Storage file system back to the NFS or Object source.
Many-to-one - You can have Sycomp Storage clusters deployed around the world accessing the same object bucket. This allows you to run your job where there is available hardware.
Configurable - You decide how often to check for source updates and how long to wait before pushing changes.
Once the cluster is deployed you can enable the Storage Scale connection to cloud object storage (COS) to automatically push your application data to Google Cloud Storage or dynamically hydrate the Sycomp Storage cluster from an existing bucket.
This makes it easy for you to use existing Google Cloud Storage data in a Sycomp Storage file system or push your file system data into the object store. You can add gateway VMs to increase the bandwidth of data movement between the file system and the object store.
Moving data to and from Google Cloud storage opens a wide range of options that can reduce storage costs, enable data sharing across applications and support bursting workloads to the cloud.
You can integrate your on-premises systems with the Google public cloud with bi-directional data flow.
Connect to an on-premises Storage Scale cluster, and NFS server or use Google Cloud Storage to share data between your on-premises data and your Sycomp Storage deployment in Google Cloud. Sycomp Storage can automatically move data to and from any NFS data source on-premises or in the cloud.
Automated data movement can be integrated with a job scheduler like Slurm or IBM Spectrum LSF to ensure data is where you need it so you do not pay for idle cloud compute time.
