Cloud Data Warehouse – Integrating Snowflake and Dimension
Investment management platforms are undergoing fundamental changes these years; vast amounts of data are collected and used for increasingly complex market and risk analysis. Further, legislation requires increasingly detailed reporting of several environmental and social factors. Whether setting up a new investment platform, or extending an existing, fundamentally new considerations are needed to ensure that your platform and service level meet your requirements in the future.
The Investment Management platform may either run locally (“on premises”) or remotely on managed infrastructure (“in the cloud”). Typically, it was installed on local servers and perhaps extended with a local data warehouse. This configuration yields total control of storage and computing power but is posing challenges for exponentially growing data. In recent years SimCorp has been extending their palette of cloud compatible applications and service offerings, informally known as SCDaaS. Last year SimCorp announced a partnership with Snowflake to utilize their platform for the Cloud Data Warehouse.
Snowflake is a cloud native data platform, with a powerful set of tools for storing and analyzing data, while integrating it with data from the marketplace. Being built in the cloud, many of the limitations of normal data warehouses are reduced or removed entirely:
- Elastic scaling of processing power and Storage
- Very high level of encryption and data protection
- Highest availability beyond 99.999%
Scaling resources on-the-fly is a key benefit of a cloud data warehouse. By elastically adjusting to the needs of your organization, you only pay for the computing power and storage you end up using. Further, management and projection of infrastructure demands proficient architects and requires valuable time being spent before implementation gets started.
SimCorp Dimension (SCD) and Snowflake run on fundamentally different infrastructure, due to the nature of the tasks they are designed for. SCD runs on an Oracle transactional database (OLTP), whereas the Snowflake ecosystem is built around an analytical database (OLAP) database structure. This means that SCD is lightning fast at fetching and updating data, whereas Snowflake is ultra-efficient in storage and security. Designed for the cloud, Snowflake is already integrated with many data sources and may be the optimal channel for data ingestion. The Snowflake Data Marketplace gives access to market data from vendors such as FactSet, MSCI, S&P, Rimes, Qontigo and many more.
Connecting SCD and Snowflake is possible using the Data Warehouse Manager and selecting Snowflake as the destination for the data model. This reduces the number of bespoke connectors and was made possible by open development standards. Utilizing Snowflake as your cloud data warehouse also means inheriting market-leading functionalities such as:
- Time-travel up to 60-days
- Zero-copy clones
- Easy data sharing
Time-travel gives a grace-period of up to 60 days (configurable) of reversing any destructive action on any object in the account, e.g. tables may easily be un-dropped. Likewise cloning of objects is near instantaneous since data is not replicated but is essentially referenced under a new object name. Providing real-time data to clients has also never been easier: Either through creating a reader account with access to authorised data or publishing data as a data provider in the Data Marketplace.
From a business perspective Snowflake integration provides numerous advantages:
- Faster and easier data integrations – No bespoke connections
- A rich Data Marketplace with validated data
- Easy roll-back due to improved data retention
Axxsys are able to provide a broad range of capabilities around both SimCorp and Snowflake. This gives us a unique insight into both platforms, how to integrate them successfully and how to leverage both to their fullest potential. We are committed to bringing our client’s platforms to the next level, so if you would like to know more about how we can support your Snowflake or SimCorp Dimensions programmes please contact us at firstname.lastname@example.org.