MessageGears
Platform Architecture
& Scale
MessageGears is the world’s only hybrid Customer Marketing Platform

We’ve been sending large volumes of messages for demanding marketers since 2011. Our ability to scale and meet the demands of Super Senders across the world stems from three key pieces of functionality, inherent in our system architecture: data access, bulk job processing, and rendering and delivery at scale.

MessageGears Segment’s ability to utilize live data in a brand’s data warehouse while not storing any additional customer data is unique in the industry, allowing for fast and powerful segmentation for our users. Most segmentation platforms fall into two categories:

  • In-Database Segmentation Platforms connect directly to a brand’s data store and allow users to segment data without needing to ship it outside of their environment. However, there are typically significant drawbacks to traditional in-DB tools, notably their need to duplicate all segmentation data into another schema. When creating segments, these tools usually create their own tables, either temporary or permanent, which adds space and strain to the data warehouse. Additionally, because of the need to execute so much custom DB logic, many of these tools have yet to adopt to modern data stores such as Google BigQuery or Snowflake
  • ESP and CDP Segmentation Tools give users an attractive interface with the promise of powerful segmentation capabilities, but these tools require users to operate off of a copy of the data itself. Brands segmenting in a CDP or ESP are fundamentally using old data that is required to fit into a rigid data model that the CDP/ESP requires. In addition to copying data, these tools do not allow brands to own their segmentation resources (as they sit in the ESP’s or CDP’s SaaS software), so performance can be slow with no recourse

MessageGears Segment takes an alternative approach to segmentation by allowing all data to live directly in the brand’s data warehouse and utilizing the power of native connections to modern data warehouses. There are two broad advantages to the MessageGears Segment architecture:

 

  1. When counting or executing segment campaigns, data is streamed out of the database and counted in our lightweight rules engine living within the Accelerator™ application. The only data stored is the metadata from the segmentation (e.g. how many recipients fall into each segment), and the time it takes to “run” the segmentation is typically 10-20% higher than the total time to return data in the query. This allows MessageGears to utilize the power of the modern data warehouse, as well as negating the need to store any Personally Identifiable Information (PII) outside of a brand’s private cloud.
  2. When activating users in a third party, MessageGears is able to break the initial recipient list into many smaller list sizes of indefinite size — this allows for parallelizing of downstream processing, such as hashing data intended for Facebook Custom Lists or Google Customer Match. The independent data sets can now be joined together for transmission to the third party, letting all of the “heavy lifting” of the activation be done within MessageGears’ cloud without any long-term storage of PII.

The most common bottlenecks for ESPs and CDPs are their rigid, multi-tenant data structures. The MessageGears system architecture (MessageGears Message and Segment) breaks through these bottlenecks by offering a schema-less customer data integration and an elastic personalization and delivery service layer.

Connecting directly to client data sources enables marketers to personalize content based upon their internal structure. MessageGears processing a schema-less structure allows for large, heavy sends to be broken into smaller batches of data and elastically distributed across our infrastructure.

Our Accelerator application initiates the message-sending process by selecting data used in the templating process from the user’s data source, and placing it into a cloud-based file of variable size according to the size of the recipient list. Although these files typically contain template-level data (email address, name, etc.), these “batches” of data are cryptographically secured via Amazon KMS encryption, and are readable only by the MessageGears cloud processing environment.

Once all of the necessary recipient data has been extracted and processed into a consumable format, an API call to MessageGears’ cloud environment begins the next step of the delivery process.

It’s worth noting that transactional messages skip bulk message processing, and are sent directly to message render and delivery. For sends to multiple recipients, upon completion of recipient upload, marketing campaign requests data flow leads into MessageGears’ cloud environment, where large data files are broken into more consumable “payloads” of data.

Operating on a single large list of unbounded size is inefficient, and would inhibit the scale and speed at which messages can be processed. This is a typical barrier for many common ESPs. However, MessageGears’ direct connection to data allows for much faster file processing in this regard. Because cloud processing is separated from the environment’s data store, MessageGears is able to disseminate recipient batches containing millions of recipients into smaller “payloads” of a more easily processable and digestible number of recipients.

Once the MessageGears cloud environment further breaks our customers’ bulk jobs into payloads, the payloads are able to be effectively and efficiently redistributed across multiple message queues that are accessible by any number of render and sending agents (MTAs) for personalization and delivery.

MessageGears’ ability to separate bulk job processing from critical render and delivery services leads to an elastic resource pool of rendering and send agents. These agents are responsible for:

  • The evaluation of the FreeMarker render of all constituent parts of the message (template, recipient, context, etc.) and appropriate recording of render errors
  • The delivery of each message on its intended channel, and recording of feedback from partner (PMTA, SMS, Push) on message delivery or failure

Our rendering agents are all constantly polling the previously mentioned payload queues for work. This technique provides virtually unlimited horizontal scaling, even at the individual job level. Our MTAs are thus designed to scale so bulk campaigns can be delivered at the same rate regardless of batch size. Because transactional messages are not processed in line with bulk messages, they are fed directly to our rendering agents to ensure processing times are always in line with SLAs.

Scalability in other areas of our platform is managed through more traditional techniques, using load balancers and auto-scaling server instances. Our analytics data leverage enterprise data streams to generate metrics on sends every minute and operate independently from all core processing.

Finally, the platform itself is distributed across multiple AWS availability zones for redundancy and configured with auto-scaling configurations to meet client demand. External monitoring and internal telemetry make sure the system is always running efficiently and in a healthy manner.