Currently, IT professionals, data analysts, and marketers come together to launch powerful cross-channel marketing campaigns within MessageGears. At a high level, all of these campaigns include data extraction with queries from the database on a starting population, processing and iterating that data in a visual segmentation, and the eventual activation of that marketing campaign based on the results of the segmentation.
What we have seen at MessageGears is that the data layer, extracting and processing the tens of millions of recipients per campaign execution, can quickly become the part of the process which takes the most amount of time. Even with modern data warehousing, extractions can take hours to count/process each recipient – making quick iteration around the blueprint process nearly impossible.
What we’ve added with the FastCache™ feature in our latest release gives the marketer the ability to quickly iterate a blueprint around a set of data to understand and specify given audiences without sacrificing the use of ‘fresh’ data. While we’ve deemed this feature to be ‘FastCache™’, there are a lot of implementation details that make a more complete solution.
The implementation details are largely split into two distinct problem areas:
- Providing up-to-date data to the marketer despite long-running queries
- Improving blueprint processing times for previously prohibitively large datasets
Providing Data Despite Long Running Queries
Modern Data Warehouses’ ability to separate compute from storage has dramatically sped up query performance, but exclusion criteria (suppression, opt-out, etc.) built into a starting population can still take queries some time to complete. With FastCache™, any qualifying action (count or extract) will save the resulting data set to the cloud bucket of your choosing. Therefore, dramatically cutting down on the number of queries to the base data store and dramatically reducing the amount of time it takes for a data set to be processed.
For example, if a query takes 10 minutes with 5 minutes of the time being the query processing and the remaining time in extracting the data, FastCache™ will automatically cut overall time in half simply by foregoing the query.
The data stored in FastCache™ is configured by the system administrator and all permissions and TTL configured by the customer so that any data held in the bucket is always as fresh as desired while making the marketer aware of the recency of the data.
While intelligent and intuitive, FastCache™ is also resilient. If for some reason your database is down, FastCache™ continues to work on the most recent successful cache. If the user attempts an unsuccessful cache refresh due to query or a database error state, an alert will be triggered and the Accelerator™ instance reverts back to the previously cached data set. FastCache™ operates with a 24-hour retention period even if you forget to save your blueprint in Segment. Once you save your blueprint, your cache stays there until you refresh it.
Additionally, all activations or message requests stemming from this data will always be processed with live data to ensure all messages are as relevant as possible – with job launches auto-refreshing the cloud storage of data so that subsequent Blueprint iterations use the freshest data as well.
For organizations using multiple blueprints with the same audience, FastCache™ intelligently reuses the cache that you’ve already stored. For instance, if you duplicate a blueprint the optimized cache is shared between the two blueprints until a change is made to query the launch variables. On the other hand, if you refresh the cache it updates immediately across all the associated blueprints. Actions like, changing a query and launching a campaign also update the cache.
Improving Blueprint Processing with Large Datasets
Aside from query timing, the blueprint processing taking place within the java application hosted behind our customer firewall was previously single-threaded in its ability to ingest and process data through the blueprint logic. This module is our Marketing Platform Accelerator™, which we have now parallelized for faster blueprint processing
With FastCache™ parallel file processing, the system administrator can dedicate as many individual cores for blueprint processing as they need. The resulting blueprint extracts will then be stored in the cloud file system aligned with the allocated resources, and blueprint extraction will be n times faster with n being the number of logical cores dedicated to processing.
For example, let’s take our 10-minute query referenced above that was reduced to 5 minutes through the use of file storage. If the system administrator then dedicates 5 cores within the application to blueprint processing the entire query and processing time drops to 1 minute – 10 times faster than the initial runtime.
Parallelization also enhances overall job performance for any external campaign or native channel activation as well. For clients processing tens of millions of recipients per execution, FastCache™ will dramatically speed time to destination.
Takeaways
We created FastCache™ with the intent that users can more easily iterate through Blueprint creation, and that campaign executions become much faster by more thoroughly relying on the resources that are provisioned for MessageGears. If you are interested in learning more about how MessageGears optimally distributes file storage across this cloud data storage to pull data for fast iteration of your visual segmentation, hit us up at sales@messagegears.com.
Danielle Briner
As the Product Marketing Manager at MessageGears, Danielle Profita Briner oversees the creation and execution of our Go-To-Market Strategy for the entire MessageGears product suite. She applies her background in content strategy and software to organize, establish, and produce meaningful content for our current and future clients.
