Deployment Models

Different ways you can deploy a Sycomp Storage cluster

Sycomp Intelligent Data Storage Platform

When deploying Sycomp Storage you can choose the amount of storage, the number of VMs running the IBM Storage Scale native client, the machine types to use and other options. Optionally you can use NFS or install the IBM Storage Scale client on your existing application nodes to give them access to the data.

An example of a Sycomp Storage deployment on Google Cloud. The Terraform tools deployed the IBM Storage Scale cluster and a set of VMs running IBM Storage Scale native client that you can use to run your application.

Figure 1 is an example of a Sycomp Storage deployment in Google Cloud. The Sycomp tools deploy the IBM Storage Scale cluster and optional set of VMs running IBM Storage Scale native client that can be used to run your AI or HPC application. In this example Google hyperdisk balanced persistent storage was used for reliability, you can choose what type of Persistent Disk you want to use.

The VMs running IBM Storage Scale Native Client and the VMs using NFS can concurrently access the data stored in the Sycomp Storage cluster. All resources are deployed within a single project.

Local SSD

You can reduce the number of Network Shared Disk (NSD) server VMs needed to reach a performance target by using Google Local SSD.

A Sycomp Storage cluster using Local SSD. All the functionality is the same except data is lost if a VM is powered off or there is a system failure.

Local SSD is ephemeral storage that offers a high number of input and output operations per second (IOPS) and low latency. To prevent data loss, Sycomp Storage leverages Storage Scale replication for Local SSD storage pools. With replication enabled the storage pool can survive the failure of a storage VM or Local SSD. In the event of a failure Storage Scale automatically creates new replicas to prevent against future failures.

Erasure Coded Protection

To protect your data deployed on storage optimized instances you can select erasure coding. Erasure coding protects your data sitting on local SSD and other ephemeral disk types available in Google cloud.

Application cluster

Whether you want to deploy multiple instance types, add a set of application nodes to handle increased year end processing, or quickly spin up a DR application cluster an application cluster is a good way to manage your compute nodes.

Figure: Example of an application cluster, once deployed this cluster is attached to a Sycomp Storage cluster containing a file system.

A client only cluster can be deployed and attached to an Storage Scale existing cluster.

A Storage Scale application cluster accesses an existing file system using Storage Scale multi-cluster. Sycomp Storage tools make it easy to maintain application clusters. Sycomp provides tools that makes it easy to integrate the use of the native client on your Slurm deployed clusters.

File System Encryption

Your deployment can be enhanced by enabling data encryption with an HA pair of VMs running IBM Security Guardium Key Lifecycle Manager (GKLM) to manage and protect your encryption keys.

An example of a basic deployment with the addition of an HA pair of key servers to support file system encryption.

Sycomp Storage automates the deployment of an HA pair of GKLM server VMs.

Cluster Expansion

You can add storage, NFS servers and clients to an existing Sycomp Storage cluster. Adding storage capacity is achieved by adding NSD servers. When you add NSD servers to the cluster each new servers gets the same disk configuration as the existing NSD servers. Storage is added in this manner to maintain balance across NSD servers. You can add NFS servers to increase capacity. Adding clients in groups allows you to customize the names and machine types of each set of clients giving you more control over your deployment.

Sycomp Storage ensures cluster expansion maintains IBM Storage Scale best practices when storage or nodes are added to the cluster by automatically deploying the correct version of software, placing disks in the correct pools and evenly distributing disks across NSD servers for optimal performance.

Intelligent Data Caching

Sycomp Storage allows you to create a copy of a remote data set that exists in Google Cloud Storage or an NFS share in a high-performance POSIX file system. The caching tool has an extensive set of features.

• Data on demand - File metadata and data is copied into the Sycomp Storage file system when a file is accessed. A directory listing in the Sycomp Storage file system fetches the file metadata, and a file open initiates a copy of the file from the remote NFS or Object store.

• Data Prefetch - To keep from wasting GPU time you can ask Sycomp Storage to prefetch a set of data before your compute job starts. You can prefetch a directory tree or provide a custom list of files.

• Read-only - You can create a read-only copy of the source data or a read-only copy that allows local modifications but never pushes the changes back to the source.

• Writable - You can create a data copy that pushes changes made in the Sycomp Storage file system back to the NFS or Object source.

• Many-to-one - You can have Sycomp Storage clusters deployed around the world accessing the same object bucket. This allows you to run your job where there is available hardware.

• Configurable - You decide how often to check for source updates and how long to wait before pushing changes.

Integration with Google Cloud Storage

Once the cluster is deployed you can enable the Storage Scale connection to cloud object storage (COS) to automatically push your application data to Google Cloud Storage or dynamically hydrate the Sycomp Storage cluster from an existing bucket.

Sycomp Storage cluster with AFM to cloud object storage (COS) configured to communicate with Google Cloud Storage.

This makes it easy for you to use existing Google Cloud Storage data in a Sycomp Storage file system or push your file system data into the object store. You can add gateway VMs to increase the bandwidth of data movement between the file system and the object store.

Moving data to and from Google Cloud storage opens a wide range of options that can reduce storage costs, enable data sharing across applications and support bursting workloads to the cloud.

Hybrid Deployment

You can integrate your on-premises systems with the Google public cloud with bi-directional data flow.

Your on-premises application can use AFM COS through Google Cloud Storage or AFM directly to create a hybrid cloud environment with Sycomp Storage deployment in Google Cloud.

Connect to an on-premises Storage Scale cluster, and NFS server or use Google Cloud Storage to share data between your on-premises data and your Sycomp Storage deployment in Google Cloud. Sycomp Storage can automatically move data to and from any NFS data source on-premises or in the cloud.

Automated data movement can be integrated with a job scheduler like Slurm or IBM Spectrum LSF to ensure data is where you need it so you do not pay for idle cloud compute time.