Imagine a scenario where a banking customer is attempting to utilize an identity verification application built on the Symbol blockchain to withdraw some much needed funds. As the anxious banking customer is waiting, the application is querying the Symbol APIs to verify whether his/her account metadata was signed by the correct entity. Suddenly, the Symbol node used for retrieving this data has gone down, and it will take around an hour to recover before chain data can be accessed again. That could lead to a very frustrating experience!
In many front-facing user applications, it’s imperative for speed and data accessibility to be one of the top priorities. With the rise of DLT applications, the need for this data is not only important from a UX perspective - but also from an auditing perspective, where transactions and (in Symbol’s case) features like account metadata have to be verified before a chain-specific action is performed. Second-layer protocols, such as IoDLT’s ANT$, our data aggregation and monetization protocol, heavily depends on Symbol API data being available.
By using Tardigrade from Storj, a distributed and affordable data storage solution, we can store Symbol’s MongoDB instance up in the decentralized cloud for future retrieval.
The importance of Symbol data being available nearly immediately from Tardigrade is a huge driver for such applications. Rather than waiting for the node to go through a recovery process that can take hours, recent data from Tardigrade can be pulled down to complete the time-sensitive operation. Meanwhile, the user notices little to no delay, even though the node is down. All the data is there, available anywhere from the world. This combination of distributed technologies proves to be an extremely powerful, secure, and practical implementation.
In an effort to streamline the tooling and workflows for developers building on the NEM and IoDLT ecosystems, we have built out an integration that makes it easy to store and distribute relevant chain information backed up by MongoDB. This greatly improves API data access for applications that use Symbol, which is imperative in DLT applications.
Using Tardigrade to store API node backups, we were able to reduce the sync time from 44 minutes to a mere 2-3 minutes - an almost 15x improvement. This means users can get data from corrupted nodes 15x faster in the event of a Symbol node restart.
Technical Dive: How NEM uses MongoDB in the Symbol Application Stack
from: https://docs.symbolplatform.com/concepts/node.html
NEM uses MongoDB in conjunction with a broker to push messages in real time to the MongoDB instance, making it chain data more accessible and queryable. The MongoDB instance retains chain data—blocks, transactions, metadata state, and generally the entire chain.
The Mongo Database is then used with some REST instances, such as this one. There is an interesting problem that can arise from this, as the chain progresses, the database will naturally get more full, making it harder to sync a full node.
On IoDLT’s IoT nodes, which run full Symbol nodes on microcomputers, this can pose a problem for running full nodes on devices with less resources, as it takes much longer to sync the chain in the case of a node shutdown.
To help solve this problem, IoDLT has built a connector for Tardigrade that periodically snapshots the chain’s MongoDB instance to the decentralized cloud, making it easy to quickly rehydrate the node’s state to any location across the globe.
Backing up a Node with the Storj MongoDB Connector
To back up our MongoDB instance, we used the publicly available and open-source MongoDB Connector for Tardigrade, located in Storj’s Third-Party directory, here: https://github.com/storj-thirdparty/connector-mongodb.
The MongoDB Connector connects to a mongoDB database, takes a backup of the specified database and uploads the backup data on Storj network. It can also restore the latest back-up of a specified database to the local storage.
ArchitectureSi from: https://github.com/storj-thirdparty/connector-mongodb
For developers who wish to try the combination of Storj and Symbol, you can try running a node on IoDLT’s Symbol devnet. This node has all the normal functionalities of a full Symbol node, but with an instance of Storj’s MongoDB backup tool integrated out-of-the-box The configuration automatically backs up your node’s MongoDB instance everyday.
To get started, get a node running in accordance in the instructions here, where IoDLT has built out the integration: https://github.com/IoDLT/catapult-devnet/tree/feat/backups
Why did IoDLT choose Tardigrade over centralized cloud providers?
The best part of Tardigrade is the service includes features by default that would otherwise cost organizations a large premium. Every file uploaded to Tardigrade stores data across statistically uncorrelated cross-geography redundancy by default, which is similar to a multi-region cloud backup architecture.
This means cloud storage backups are always accessible, and a single outage won’t cripple our operations like it would in a centralized cloud storage architecture. Tardigrade backups are available everywhere, and with hot availability, developers can quickly rehydrate their MongoDB instance from anywhere in the world.
Tardigrade and MongoDB help to streamline blockchain development workflows, making the storage and transfer of Mongo dataset snapshots much easier. Global accessibility, means that data is distributed everywhere, making syncing across geographies seamless.
Looking Forward - The Potential of IoDLT, Storj, and Symbol
With the release of this first tool, the relationship between IoDLT, Storj, and Symbol has only just begun. We see a number of interesting future avenues for collaboration between Symbol’s DLT, IoDLT’s products, and the Storj Network. IoDLT’s flagship products, such as Axon, our multipurpose IoT device, and ANT$, both can utilize the auditing capability of Symbol and the affordable storage that Tardigrade offers.
Due to its distributed and parallel architecture, we think that the decentralized Storj Network - and the Tardigrade Service is the “missing piece of the puzzle” for DLT applications. Tardigrade makes it very easy to store data across thousands of uncorrelated nodes in a more reliable, resilient, and responsive manner. Symbol ensures immutable, foolproof record keeping auditing capabilities in a distributed and streamlined manner. We see the following areas as potential for future collaboration between Storj/Tardigrade and Symbol:
Symbol can be used for auditing file storage, retrieval, and air-tight permissions - Symbol contains built-in features, such as Account Restrictions and Multisig Groups to allow for access control to be easily facilitated on-chain. The natural accounting functionality of the ledger can serve as an audit mechanism, where whoever accessed sensitive files can be traced back to the point of upload.
Further Applications Using Tardigrade, Symbol and IoDLT
An exciting, but experimental idea is using Multisig “multicast” groups for sharing access keys. Specified users within a multisig account (essentially cosignatories) can receive Storj access keys for accessing buckets on Storj. Symbol would audit this transfer, and who is able to access the bucket can be represented on-chain.
All of this would be done mostly through a theoretical “StorjTransaction” (Symbol plugin). As Symbol allows for the extension of its functionality through extensions and plugins, a homogenous link to a configurable Tardigrade instance could be specified directly within the core server.
This type of integration would allow for IoDLT’s multipurpose IoT device, Axon, to store videos and pictures on the blockchain in a much more efficient way. This ability to store more comprehensive data from IoT has endless applications across healthcare, supply chain, and security use cases.
If you are interested in learning more about IoDLT and the work with the Symbol Stack, MongoDB, our own products, and Storj Labs - reach out directly at: [email protected]
About IoDLT
Founded in 2018, IoDLT (Internet of Distributed Ledger Technology) utilizes two disruptive technologies – Internet of Things and blockchain – to provide seamless, secure, and scalable B2B solutions. IoDLT brings security to small and large businesses alike, without compromising user data privacy and user-to-business interactions. Their technology’s application spans a wide range of industries, namely healthcare, agriculture, supply chain, and energy metering.
Alongside providing business solutions, IoDLT envisions a future run by embedded devices. Securing those devices will become imperative to the operations of any business. IoDLT deploys proprietary and affordable IoT to blockchain protocols to secure the devices of the world
