At-Scale Data Movement,
How To Do It Right

Zettar Inc. (Zettar) builds and delivers a unified, simple, scalable, efficient, and versatile at-scale data movement manager. The product is ideal for distributed data-intensive engineering and science workloads such as for genomics, life sciences, Oil & Gas, AI, machine learning, transporting data for large-scale IoT deployments, autonomous vehicle fleets, smart cities, EDA, Media & Entertainment Studio post-production, light sources (large lasers), accelerators, large telescopes.  It is excellent for tackling today’s ever-growing edge to core/cloud use cases.  It can also dramatically improve the Environmental, Social, and Governance (ESG) benefits of composable/disaggregated infrastructures.  It does so by endowing the key devices DPU/IPU with a wide spectrum of built-in data movement capabilities , making them far more effective.

The Zettar team has rich first-hand solution architecture experience in helping tier-1 customers in the biopharmaceutical, Oil & Gas, Media & Entertainment Studios, and supercomputing centers in different countries. As a result, even as a software company, the Zettar engineering team has a deep and comprehensive understanding and expertise of the entire infrastructure stack, storage, computing, and networking (including network security).  For example, the Zettar Engineering team members are active contributors to the most modern storage freeware benchmark elbencho.

Furthermore, from the engagement supporting the highly ambitious data movement requirements (>= 1Tbps point-to-point by 2024) of Linac Coherent Light Source II (LCLS-II), a premier U.S. DOE Exascale Computing preparation project hosted at the SLAC National Accelerator Laboratory in Menlo Park, California, all members have gained extensive experience applying the U.S. DOE Exascale Initiative’s “co-design” principle – integrated consideration of storage, computing, networking, and concurrent software for optimal performance.  Thus, Zettar is a genuinely engineering-centric software company.  Hence, working with Zettar will help your business to gain such valuable experience as well.

Zettar has been focusing on at-scale data movement since 2014. The company is supported by its revenue. A few engineering initiatives are funded by the U.S. National Science Foundation (NSF) and U.S. Department of Energy, Office of Science.

Since early 2015, Zettar has been engaged to support the the highly ambitious data movement requirements of Linac Coherent Light Source II (LCLS-II), a premier U.S. DOE Exascale Computing Initiative (ECI) preparation project hosted at the SLAC National Accelerator Laboratory in Menlo Park, California. As a result, all engineering members have gained extensive experience applying the U.S. DOE ECI’s “co-design” principle – integrated consideration of storage, computing, networking (including network security), and concurrent software for optimal performance. Thus, working with Zettar will help your business to gain such valuable experience as well.

Foremost,  we strongly recommend getting all infrastructural stacks ready, storage, computing (servers, physical and/or virtual), and networking (including network security, e.g. firewall).   At-scale data movement is neither a software-alone nor network-alone endeavor.  Furthermore, an application can only run so well as its environment (and its settings, which depend on the environment) allows it to.

Never think like this: “Oh, let’s just install the software and see how fast it runs!”  That’s not for professionals!

Please Contact Sales.  Zettar’s Tech Sales team will be in touch and provide you the necessary info.

Zettar zx is the world’s only software data mover that can tackle the two major data movement types, bulk transfer, which deals with already stored, and existing data, and append streaming, which deals with live, fast-growing data, most likely machine generated.  If your need falls into either category, and it is not strictly an end-user oriented data movement needs, talk to us.  We will discuss and ensure zx is a good fit for you.

Once you are here, please Contact Sales.  Our Tech Sales and Solution Architect will be in touch and help you further in your planning and decision-making.  Of course, we invite to review the rest of this documentation.  The information should be helpful as well.

Zettar delivers a unified, at-scale data movement manager, zx, that provides top performance, simplicity, manageability, scalability, and breakthrough economics. zx seamless integrates four distinct products.  As such, it enables you to tackle almost all conceivable data management tasks that involve at scale data movement: data protection, vaulting, moving, and synchronization.

The software runs on standard server hardware (both x86_86/amd64 and aarch64 CPUs supported) with Linux OS and delivers 10X or more the performance than the typical commercial or freeware counterparts.  Zettar does not employ marketing hot-air – we don’t need to.  Please see the Company, Publications for publications from various super-computing centers, U.S. DOE ESnet, and prestigious business publications.

The Zettar zx has some very important core attributes.  It is:

• Purposely designed for moving data at scale data movement – distributed, scalable, and running as a daemon (i.e. it is not a CLI utility)
• A hardware agnostic solution
• A solution that supports both POSIX-compliant file storage and AWS S3 (including true compatibles) object storage

Hardware technology changes over time; a software data mover solution should accommodate such changes, which means that it must be able to run on any standard hardware platform.  zx was designed to run on any standard x86_64/AMD64/aarch64-based server hardware.  It works with conventional HDDs or the latest SSDs.  zx also runs natively in a public compute cloud or works directly with a public storage cloud.  zx eliminates the cost overhead of expensive specialized hardware and allows you to benefit from advances in technology without suffering the pain of forklift upgrades to next-generation architecture.  Furthermore, since zx can be embedded in DPUs/IPUs, which also enable to “upgrade” a server without fork lifting, so there are more than one way to do it.

zx is ideal for transporting massive amounts of data for the following:

• Life Science research data collection, replication, and exchange – Next Generation Sequencing (NGS), bio-imaging, cancer research, molecular diagnosis, structural biology, and bio-informatics
• Oil & Gas exploration data transportation – among facilities or between on-prem and cloud
• Large scale NAS tech refresh – same vendor, different vendors
• File system migration – between the same type (e.g. IBM GPFS) or different filesystems (e.g. from IBM GPFS to BeeGFS), or between NAS and a parallel filesystem
• Data migration between file and object storage (on-prem or public) – e.g. from a NAS to AWS S3
• HPC – light source and nuclear accelerator detector data off-site processing, camera data of large telescopes off-site for processing and storage; climate change simulation, computational physics, earthquake studies, space research, simulation, intelligence
• In-vehicle data collection for fleets of autonomous vehicles – Transporting collected data to data center and/or cloud
• AI, Machine Learning, and work flows involving GPU acceleration – both on prem and in the cloud
• Media and Entertainment – pre- and post-production workflow acceleration, content delivery, digital asset replication and disaster recovery

More are feasible.  Please Contact Sales. Our Tech Sales and Solution Architect will be in touch and help you further in your planning and decision-making.

Absolutely!  zx can do incremental backup jobs and carry out “snap diff” (i.e. comparing snapshots provided by the storage system).  Please also see this 2 min 29-sec video “Zettar zx’s feature: rsync on steroids, 

zx is an easy-to-deploy data mover solution that is configurable to fit your environment, giving you complete deployment flexibility.

• Hyperconverged deployments leverage your existing compute infrastructure while eliminating your data transfer setup footprint and reducing power and cooling costs.
• Pooled storage deployments are ideal when you want to use separate storage and compute infrastructure for application isolation, performance, or scalability.
• Public cloud deployments allow you to realize the promise of truly elastic computing by running zx on public cloud server instances.

The software runs well in physical servers, virtual machines,  containers, and devices such as DPUs/IPUs.  Which one to choose depends on the use case.

• zx provides flexibility, ease of deployment, and resiliency, whether on-premises, in a hybrid configuration, or entirely in the cloud for on-demand scalability.  zx’s very small footprint makes it embeddable as well.
• zx is an at-scale data movement data mover that provides the freedom to choose the environment best suited for your application based on performance, scale, and economics.
• It targets 1) Red Hat Enterprise Linux 7.x, 8.x, and 9.x or a free rebuild like Rocky Linux, Alma Linux, or Oracle Linux 2) Ubuntu 20.04LTS, and newer.
• zx supports x86_64, amd64, and aarch64.

Being a unified at-scale data mover, Zettar zx has the following key strengths:

1. Simplicity – from the simple installation, configuration, and operation, to its rich integrated functionalities, Zettar Engineering has strive to make every aspect as simple as possible.
2. Scalability – among all data mover software applications, it’s one of the only few (all have been created with U.S. DOE Office of Science’s support) that is truly scale-out capable, not with some “cluster workload managers”. This fact again contributes to Zettar zx’s simplicity.
3. Efficiency – we have not run into any other data mover, free and commercial, that exhibits the same level of efficiency as Zettar zx.  These are not empty words. We have enough deeds to prove (About Zettar, Publications). 
4. Versatility – it can tackle numerous seemingly different use cases simply, consistently, and efficiently.

The tight integration also provides highly consistent usages across different products as well.

Each instance of zx or a cluster of zx instances can support multiple users, and each has an exclusive data area for reading and writing. Complete visibility of a task is only available to the owning user. 

Absolutely!  As long as both ends run zx.  

Absolutely!  Federated learning involves multiple collaborative institutions.  Each must carry out computing work on-prem. Even within the same institutions, data sources and where computing is carried out are almost always different.  For example, the lab housing next-generation sequencers (NGS) for genome sequencing is usually a building with strict environment (including noise) control. The researchers working there are definitely not working on IT tasks.

The data generated per day, depending on the number of NGSes, can easily exceed 100 TB!  Moving so much data to a nearby storage and computing facility is non-trivial. 

Federated learning depends on on-site (or edge) computing. Given the above, moving data is still an essential part of the overall workflow. zx excels in such situation as well.

Unlike bulk transfer, which deals with already stored, existing data, append streaming tackles a far more challenging situation: data that’s live and fast-growing (e.g. at a rate of 200 Gbps; 1 Gbps = 1000 Mbps; 1 Mbps = 1000 kbps), likely generated by machines such as high-output instruments.

It is not only new but also unique.  Zettar is the inventor of this technology. No other data movers, commercial and free, have it.

They are not the same thing at all. 

1. The typical video streaming is actually bulk transfer in disguise. Basically, existing “assets” stored on CDN edge servers are divided (logically) into multiple “slices” in response to a “streaming request”.  The size of slices depends on the perceived bandwidth of the requester. So, in a way, this is like bulk transfer in chunks of an existing file.
2. Append streaming, on the other hand, must deal with live and fast-growing data.  As an analogy: it is very trivial to copy an existing spreadsheet file from one folder to another.  But it’s much more challenging to copy a video that is still being created!

Furthermore, to give you an idea of how fast the “fast-growing” is. With LCLS-II, the data growth rate is 800 Gbps. A typical video streaming data rate is around 20 kbps.  The former is four orders of magnitude more intense than the latter.

Yes. Append streaming enables a real-time backup of fast-growing, live data such as output by high-speed instruments. For example, the camera of a photon-counting CT can produce 1 TB of fresh image data in a few minutes.   

As an example, assuming it takes a CT 4 minutes (4×60 sec = 240 sec) to produce a 1 TB (=1024 GB; 1 B = 8 bits) fresh image file, the data growth rate = 1024 GB/240 sec = 2.84 GB/sec  = 34 Gbps. zx append streaming can comfortably handle such a growth rate.

No, it’s not.  Open source doesn’t fit our business model.

On a per instance basis:

• An FTP server application typically runs on a single computer. Even being confined to running in this manner (i.e. no scale-out), zx is usually 10X or faster.  rsync, scp, sftp, and robocopy are all end-user-oriented CLI tools. Even if they are used by experienced users, the same range of speed-up has been observed in real-world usages.  Once the scale-out capability of zx is leveraged, such CLI tools will be left far behind.  Even if a threaded and cluster-capable FTP application is used (e.g. GridFTP), zx still holds both efficiency and performance advantages by a large margin (> 50%) over a wide file size range.  FYI, at the Olympic level data mover competition, Supercomputing Asia 2019 Data Mover Challenge, Zettar beat out the Globus author team of GridFTP (slide 7). 
• Zettar zx supports both file and AWS S3 (including true compatibles) object storage. None of those CLI tools do so either at all or fully.
• Most importantly, once zx is set up, it offers operational simplicity and manageability these CLI tools can not match. For example, zx provides both built-in Web UI and API (with an open-source Python SDK) which enables simple yet powerful automation. Furthermore, it offers advanced workflow management features such as check-point, restart, sophisticated bandwidth throttling, and multi-level parallelism that works with storage, computing, and network resources.  A critical benefit is that it’s proven network latency insensitive.  None of those popular CLI tools can offer all such benefits.

s3cmd is a CLI end-user oriented tool running on a single computer.  Even being confined to run in a single computer, zx is usually anywhere from 10X or more faster per instance.  Once the scale-out capability of zx is leveraged, s3cmd will be far behind. 

When properly configured, rclone can attain a good transfer rate when the file sizes are suitable for its processing.  But it lacks sophisticated data management capabilities, with only an experimental UI (as of this writing), not scale-out capable, and not capable of preserving file attributes either when files are transferred to a cloud storage (including AWS S3). 

zx’s benefits over other CLI tools are applicable here as well.  The ease-of-use aspects mentioned previously also apply in this case.

The AWS’s DataSync page states, quoted “AWS DataSync is an online data transfer service that simplifies, automates, and accelerates moving data between storage systems and services.”  and some functionalities. 

In brief, “everything that it can do, Zettar zx can do more efficiently and much simpler. Many feats that zx can carry out routinely, such as embedding in any IPUs/DPUs, it’s impossible for DataSync.”  Furthermore, even your company is a heavy use of AWS, there are still many data movement use cases that AWS cannot get involved at all but zx can still help you.

Such applications are limited to the traditional end-to-end data transfers – a very narrow scope.  In addition, as a rule they are not scale-out capable. Some may claim that they have patented data transfer protocols, but their results do not jibe with facts.  Also, even being confined to run in a single computer, zx is usually anywhere from 10X or faster per instance.  Once the scale-out capability of zx is leveraged, such tools cannot even match.  Most importantly, once zx is setup, it is much easier to use.  See these capability videos above.

Most of them also have very limited scope – typically only capable of dealing with the traditional end-to-end data transfers.  Almost all well established data movers were introduced around 2000 – that’s 20 years ago. They did their jobs.  But the world’s exponential data growth started in 2016, per Intel DCG EVP Navin Shenoy.  Zettar zx is designed from the ground up to address this challenge. The problems when these well-known data movers came into being and the problems we address demand very different approaches and architectures. You are invited to review this 1min 48 sec video, Learn the water transport analogy, up on this page.

Zettar started off in 2015 by supporting a premier U.S. DOE Exascale Computing project, Linac Coherent Light Source II, which has highly ambitious data movement requirements (not just the traditional transfers!). The Zettar Product Brief up on this page has some details. Striving to meet such demanding requirements ever since, by 2018, zx has been able to attain excellent outcomes in the following production trials (not demos!) and international competition:

1. In September 2018, using a modest test bed, under a hard 80Gbps bandwidth cap, with full encryption and checksumming, zx transferred 1PB in 29 hours over a 5000-mile loop provided by the U.S. DOE Energy Sciences Network (ESnet). 94% average bandwidth utilization was achieved.
2. In March 2019, the 2-person Zettar team competed in and became the overall winner of the grueling 2-months long SCA19 DMC, this is the Olympic competition for data mover software.  The six other participants are elite national teams (slide 7).  Over two successive SCA DMCs (2019 and 2020), Zettar is still the only Overall Winner with its record unbroken.
3. In October – early November, 2019, working with the Interdisciplinary Centre for Mathematical and Computational Modelling (ICM)– University of Warsaw (Poland), A*STAR Computational Resource Centre (A*CRC, Singapore), the joint-effort succeeded in achieving a historical 1^st data transfer production trial over the then brand new Collaboration Asia Europe-1 (CAE-1)  100Gbps network across a vast distance of 12,375 miles.

Please note that the above are all published by government agencies and supercomputing centers. 

Sure!  The U.S. DOE Office of Science over the years has invested in the creation and development of a few heavy duty data movers (including Zettar zx).  Two national laboratories that have been funded are FNAL and BNL.  Together, the two national labs have created the MDTM project. The researchers there have conducted comparison with other DOE funded data movers and the commercial IBM/Aspera and reported their findings in the MDTM Project Review Documents

According to the the report. IBM/Aspera is 3.9X slower than mdtmBBCP (slide 50 & 51).  But the author team of the MDTM project was beaten out by the Zettar team at SCA19 DMC.  Enough said?

At the 2 month-grueling, Olympic Competition of data movers, Supercomputing Asia 2019, Data Mover Challenge, the Zettar team became the only Overall Winner awarded, beating out 6 other elite national teams, including the Globus GridFTP author team From ANL and the University of Chicago and The FANL’s BigData Express (which uses MDTMFTP and MDTMBBCP) author team by about 50% of margin. With the latter, when a horizontal scale-out setup is used, it will be lagging behind even more.  

From October – November 2020, the U.S. DOE Energy Sciences Network (ESnet) evaluated Zettar zx as reported in Zettar zx Evaluation for ESnet DTNs.  Fig 1 of the report shows that over a wide range of file sizes, Globus GridFTP is 2 – 5 times slower per node. BigData Express is not even in the same league as GridFTP and zx, since neither MDTMFTP nor MDTMBBCP is cluster-oriented.  When the employed storage system enables zx’s linear scalability to kick in, even the cluster-capable GridFTP will be left far behind.

Briefly:

• Both Globus GridFTP and BigData Express lack the efficiency to tackle fast modern data growth. 
• Furthermore, since both are based on the traditional client-server design, so they are good only for “man-in-the-loop” type of data transfers.  For example, neither would be awkward and of limited use in the very common large-scale file system migration scenarios.
• Zettar zx however, tackles many use cases competently, fully automated or man-in-the-loop.

Yes, zx has the following attributes:

• zx is symmetric.  Any instance can send and receive with the other zx instance, even concurrently.  In other words, its architecture differs from the traditional client-server model, for at least 2X simplicity
• Its transfer performance is highly predictable – with sufficient computing power and network bandwidth, usually it attains 80% or more of the measured storage throughput
• It is truly network latency insensitive – this has been proven repeatedly over a 5000-mile ESnet loop from 2015 – 2019, and many other high-profile international events,  e.g. SCA19 DMC and Poland-to-Singapore data transfer trial over CAE-1.
• It is the world’s only data mover software capable of tackling both bulk transfer and append streaming. Both are forwardable too.

It is the most modern scale-out capable data mover software funded in 2019 by the U.S. DOE Office of Science.

zx employs unconditional end-to-end checksum.

zx provides linear scalability – the throughput increases linearly as the number of zx instances, assuming

1. The storage throughput available to each instance is the same
2. The available network bandwidth can accommodate the collective throughput
3. Sufficient computing power is available to each zx instance
4. Firewalls are not hindering the desired data rates.

zx supports both POSIX and AWS S3 (including true compatibles).

This is extremely easy and only takes a few minutes.  Please see the following:

1. zx is a data mover. It runs on a compute node (aka data transfer node, or DTN).
2. The compute node should be configured as a NFS client, preferrably running a new enough OS that allows the use of the nconnect NFS option.
3. Note that you can import multiple NFS shares and zx can work with all of them concurrently if they are mounted under the same root. This is true both for reading and writing.

zx can use the available bandwidth anywhere from Mbps to hundreds of Gbps. Typically, with modern hardware, assuming sufficient storage throughput available, a decent CPU model, fully populated CPU memory channels, enough available network bandwidth, and normal average file/object sizes (>= 4MB), a single data transfer node running zx can comfortably push/pull 70+Gbps or beyond. At the SCA19 DMC, even with the sub-optimal setup and a 2TB dataset consisting mix-sized files, Zettar was still able to reach this level.  See the official announcement. So, using two nodes at each end not only have the potential to go beyond 100Gbps but also provide high-availability (HA) at the same time.

zx is insensitive to network latency.  This has been proven

• In numerous production trials over a 5000-mile 100Gbps “loop” provided by the U.S. DOE Energy Science Network, for example, please see this September 2018 production trial.
• At a highest level international data mover competition: SCA19 DMC
• The historical 1st Poland-Singapore data transfer production trial over the CAE-1 100Gbps network across 12,375 miles. 

Zettar Engineering has also done many transfers at much lower bandwidth internationally, e.g. 80Mbps from Europe to N. America.

zx compares the differences between two file data sets and transfers only the difference (i.e. delta).  Thus, other than just bulk transferring/streaming data, zx can be used for efficient incremental replication.

zx can be used for large scale replication tasks, such as tech refresh of many NASes, large parallel file system storage pools.

zx offers standard based TLS encryption with various ciphers available to fit your requirements.

Zettar zx employs the open, industry standard JSON Web Token (JWT) for representing claims securely between two parties.  The authentication of JWT uses private/public keys.

Getting started with Zettar is easy.   Please fill out the Contact Sales form.

Zettar software package (a single RPM or DEB) includes everything you need for high-performance, scale-out data transport. There are no additional license fees for standard enterprise file-based features.

Zettar supports annual subscription and perpetual license, both come with white-glove support.  The 2^nd model may need you to purchase an annual Software Maintenance and Support Agreement, which is mandatory for the 1^st year.  The software is licensed on a per-node basis.  Given its proven efficiency, even a large site usually needs just a few licenses.

As long as there is an active Software Maintenance and Support Agreement in place, Zettar license holders are entitled to all software fixes and product enhancements as part of the base product (zx-File) license.  Please note that from time to time, Zettar may introduce new products (e.g. zx-Object, zx-Single-Site-Mode, zx-Append-Streaming) that are integrated into zx and are available for purchase.  Such new products must be activated via additional licenses and would not be included in the base product license entitlement.

Zettar understands that your usage patterns and needs may change from time to time. So you can let your license expire if it’s of the annual subscription type. 

zx’s built-in WebUI About page, License tab will show you the respective license expiry date of what you have licensed.  Also, if your license is of the annual subscription type, zx will stop working post the expiry dates.  With perpetual licenses zx always works.  But we encourage you to keep the Software Maintenance and Support Agreement renewed promptly to keep your base software always current and benefit from our professional support.

You can reinstate either the software license or the Software Maintenance and Support Agreement or both at any time after expiration by simply contacting us via the online form or email.  Written requests are necessary for both sides’ records.

We are afraid for what the software designed for, free download doesn’t make sense and is illogical.  At-scale data movement demands foremost proper infrastructure preparation and readiness.  It is never a software alone endeavor!  Even you have the software, but your infrastructure is not conducive to what you desire to achieve, the software cannot magically make things happen.

If your organization is serious in embarking such a project, we advise you contact us to discuss. Then, we can work with you on an evaluation – a typical one, even remotely with a non-domestic (in U.S.) customer, generally it takes about 3 – 4 1-hour sessions to gain sufficient experience for your decision making.