1. Data Flows

Understand the flows of data for a typical connected integration.

What you’ll learn

In this section of the Connected Integration Blueprint (CIB) you’ll learn about:

  • The core data entities of the Marketplacer platform
  • The core relationships between those entities
  • The core flows of data in and out of a Connected Integration
    • (Between Markertplacer and your eCommerce solution)

Core Entities

Marketplacer is a large product, comprised of many entities, so to describe them all here in detail would be counter to the nature of this playbook. Instead, we will focus on the core (or primary) entities that will be of interest to you when integrating into Marketplacer.

The table below describes the core entities we’ll be discussing in the rest of the CIB.

EntityAliasPurposeContext
AdvertProductPrimary container for Products that a Seller wishes to sell on the marketplace. These products would be referred to as 3rd party products when sold along side an operators own 1st party products.Products
Variant Describes how the Advert can vary, e.g. different sizes or colors. An Advert must always have at least 1 variant. It is at the variant level where attributes such as: SKU, Bar code, Inventory are located, and ultimately it is the variant that is purchased by the customer.Products
Order An Order is the “marketplace-level” object that acts as the container for a customer’s entire purchase. An Order in Marketplacer will always have at least 1 Invoice attached to it, (1 invoice per seller)Orders
InvoiceOrderAn Invoice is the part of the overall customer order that an individual Seller has to fulfill. An Order will always have at least 1 Invoice (1 Invoice per Seller participating in the Order). From a Sellers perspective the Invoice is the Order.Orders
Line Item Line Items represent the products purchased by the customer. Line Items have a direct line of sight back to the Variant. I.e. when a variant is purchased it is then represented as a Line Item. Line Items are attached to both the Order and the Invoice.Orders
Refund Request A Refund Request models the stages that both Cancellation (pre-dispatch) and Refund (post-dispatch) requests go through. A successful Refund Request will result in an Invoice Amendment.Orders
Invoice AmendmentRefundRepresents an adjustment to an Invoice. Usually generated after a successful Cancellation or Refund Request.Orders
Shipment Represents the dispatch of 1 or more Line Items, Sellers are typically responsible for the creation of shipments.Shipments
Shipped Item You can choose to split shipments, so either individual line items on an Invoice, or even specific quantities of the same line item. This is represented by a Shipped ItemShipments
SellerRetailersSellers are 1 of the primary participants on the marketplace and are responsible for the listing of (3rd party) products for sale.Sellers
Remittance The marketplace operator is responsible for taking payment for orders, remembering that orders can be comprised of products from multiple sellers. This is referred to as the Merchant of Record. At some point however the Sellers participating in marketplacer orders will need to be paid (by the marketplace operator), this payment is referred to as a Remittance.Sellers
TaxonCategoryRepresents where the product sits in the categorization hierarchy, e.g. Computers -> Storage -> SSH Hard Drives. In this example we have 3 Taxons, with Parent / Child relationships between each.Categorization
Option TypeAttributeOption Types refer to the the way a Variant can be defined. So common Option Types would be color and Size. Option Types will then have 1 or move Option Values, which contain the range of values that it is possible for the Option Type to have.Categorization
Option ValueAttribute ValueThe values that an Option Type can have. In the example where we have an Option Type of Color, possible Option Values would be: Red, Green, Blue etc.Categorization
PrototypeAttribute Profile or GroupPrototypes are the glue between a Taxon and the Option Types that Taxon has. So we assign Option Types to a Prototype, and then assign a Prototype to the Taxon. This means we can assign that same defined Prototype to any number of other Taxons.Categorization

Entity Relationships

We’ll start by looking at each of the entity relationships grouped by context, this is easier to digest at first, and is arguably the most useful view anyway!

Products

Product Entities

  • An Advert must have 1 or more Variants
  • A Variant must have 1 Advert

Orders

Order Entities

  • An Order must have 1 or more Invoices
  • An Order must have 1 or more Line Items
  • An Invoice must have 1 Order
  • An Invoice must have 1 or more Line Items
  • [Optional] An Invoice can have 1 or more Invoice Amendments
  • [Optional] An Invoice can have 1 or more Refund Requests
  • A Line Item must have 1 Order
  • A Line Item must have 1 Invoice
  • A Refund Request must have 1 Invoice
  • An Invoice Amendment must have 1 Invoice

Shipments

Shipment Entities

  • A Shipment must have 1 or more Shipped Items
  • A Shipped Item must have 1 Shipment

Sellers

Seller Entities

  • [Optional] A Seller can have 1 or more Remittances
  • A Remittance must have 1 Seller

Categorization

Categorization Entities

  • [Optional] A Taxon can have 1 Prototype
  • [Optional] A Prototype can have many Taxons
  • [Optional] A Prototype can have many Option Types
  • [Optional] An Option Type can have many Prototypes
  • [Optional] An Option Type can have many Option Values
  • [Optional] An Option Value can have 1 Option Type

Extended View

Domain Entities


We have just listed the cross-context relationships below:

  • [Optional] A Seller can have 1 or more Adverts
  • An Advert must have 1 Seller
  • [Optional] An Invoice can have 1 Remittance
  • A Remittance must have have 1 Invoice
  • [Optional] An Invoice can have 1 Shipment
  • A Shipment must have 1 Invoice
  • A Line Item must have 1 Variant
  • [Optional] A Variant can have 1 Line Item
  • [Optional] A Line Item can have have 1 Shipped Item
  • A Shipped Item must have 1 Line Item
  • [Optional] A Variant can have many Option Values
  • [Optional] An Option Value can have many Variants
  • [Optional] An Advert can have 1 Taxon
  • [Optional] A Taxon can have many Adverts

We will be referring back to this model throughout the rest of the CIB.

Data Flows

Before talking about the specific individual Data Flows, it’s worth revisiting the high level Product/Order flow of a typical marketplace:

End to End Flow

  1. Sellers will list products on the marketplace
  2. Seller products (aka 3rd Party Products) will need to flow through to the eCommerce platform
  3. Customers will browse products on the front end (3rd party and possibly 1st party)
  4. Customers will place an order on the eCommerce platform (payment is taken here)
  5. Orders (for 3rd party products only) are passed through to Marketplacer
  6. Sellers will pick up their orders and process

Connected Integration Context

For the purposes of this section, and this blueprint as a whole, we will be focused in on the following data flow context:

Connected Context

Therefore in this section we are only going to talk about the movement of data between Marketplacer (MP) and the Connected Integration (CI) and vice versa. In later sections we will however discuss:

  • Which APIs and Webhooks you can use to move this data (Section 2 - Integration Points)
  • Approaches to how you could compose your integration (Section 3 - Architectural Patterns)

It’s also worth clarifying the following terms when we talk about the flow of data:

  • Information Provider
  • Information Consumer

Information Provider

The Information Provider (referred to as just the “Provider” from now on) is the source of the data to be moved. E.g. If a product is created by a Seller in Marketplacer and that Product needs to propagate to the Connected Integration, then:

  • Marketplacer (MP) is the Provider
  • The Connected Integration (CI) is the consumer

Information Consumer

The Information Consumer (referred to as just the consumer from now on) is the target of the data to be moved. We can refer to the example given above to understand the role of the consumer.

Our context can be summarized as follows:


Provider Consumer


An expanded view showing the high-level flow of the entity contexts is shown below:


What you should sync

Product Flows

Let’s remind ourselves of the product entities:

Product Entities

At a simple level Advert and Variant data will need to flow from Marketplacer (MP) to the Connected Integration (CI). Less common, although certainly valid for some use-cases, Advert and Variant data may flow back from the Connected Integration to Marketplacer. Let’s explore some specific product use-cases in more detail:

ProviderConsumerUse CaseDescription
MPCIProduct CreatedWhenever a seller creates new products in Marketplacer these will need to flow through to the Connected Integration.
MPCIProduct Updated (Ad Hoc)Existing products will be updated for various reasons, these updates will need to flow through to the Connected Integration
MPCIProduct DeletedExisting products may be deleted by the Seller, and will therefore no longer be available for sale, this will need to flow to the Connected Integration
MPCIProduct Off-linedSellers may choose to “offline” a product if they no longer want to take orders for it, this is different from deletion as they may want to bring the product back online at some point.
MPCIInventory UpdatedInventory updates occur at the variant level (this is where we hold Inventory Position). This is a unique product update use-case in that the cadence of updates can be high, there are also potentially larger implications for operator and seller if this data is not accurately reflected across domains.
MPCIPricing UpdatedThis is essentially just another product update, but has been called out specifically given the potential for regular of updates, as well as he criticality of that data.
MPCIAdvert VettedOperators can choose to enable Advert Vetting, meaning that products are not automatically placed in an online state and made available for sale. If an Operator chooses to vet a product using the Marketplacer Operator Portal, that update should propagate to the CI.
CIMPAdvert VettedSame background context as above, but Operators can choose to vet products using the Marketplacer API, in which case this data is flowing form the CI to Marketplacer.
CIMPProduct EnrichedThere are some use-cases where operators have specific requirements around how products are listed, in some instances this means that operators enrich seller product listings using the Marktplacer API to adhere to their requirements.


Order Flows

Let’s remind ourselves of the order entities:

Order Entities

We have chosen to include Refunds as part of the Order context, and as such we’ll cover the data flows here. It’s worth noting 2 points however:

  1. Refunds are usually tackled later in the development cycle
  2. You could arguably split out Refunds into a separate context - feel free to do so if this fits your use-case.
ProviderConsumerUse CaseDescription
CIMPOrder CreatedOne of the core flows of the entire integration, the movement of order data through to MP, note that Invoices are created at essentially the same time as the order, therefore we do not have a separate Invoice Created flow.
CIMPOrder UpdatedOrder updates from the CI to MP are rare, indeed the only reason it hs been captured here is to call out that it almost never happens. You may want to refer to some of the refund flows for use-cases around cancelling and refunding orders.
CIMPRefund Request CreatedRefund Requests cover both order cancellations (pre-dispatch) and refunds (post dispatch). Requests can be raised via the CI (using the Operator API), and propagate to MP.
CIMPRefund Request UpdatedRefunds (and Cancellations), go can go through 1 of 2 workflows depending on your Marketplacer configuration. Either way, in both cases updates to the Refund Request can flow from the CI to MP. Requests can be raised via the CI (using the Operator API), and propagate to MP.
CIMPRefund Request ApprovedFinalizing the refunds workflow can result in the approval of the refund request. Requests can be raised via the CI (using the Operator API), and propagate to MP.
CIMPRefund Request DeniedFinalizing the refunds workflow can result in the denial of the refund request. Requests can be raised via the CI (using the Operator API), and propagate to MP.
MPCIRefund Request CreatedRefund Requests cover both order cancellations (pre-dispatch) and refunds (post dispatch). Requests can be raised using the Seller and Operator Portals in MP, if this occurs then this data can flow back from MP to the CI.
MPCIRefund Request UpdatedRefunds (and Cancellations), go can go through 1 of 2 workflows depending on your Marketplacer configuration. Either way, in both cases updates to the Refund Request can flow from the CI to MP. Requests can be raised using the Seller and Operator Portals in MP, if this occurs then this data can flow back from MP to the CI.
MPCIRefund Request ApprovedFinalizing the refunds workflow can result in the approval of the refund request. Approvals can be raised using the Operator Portal in MP, if this occurs then this data can flow back from MP to the CI.
MPCIRefund Request DeniedFinalizing the refunds workflow can result in the denial of the refund request. Denials can be raised using the Operator Portal in MP, if this occurs then this data can flow back from MP to the CI.
MPCIInvoice UpdatedAlong side the creation of the order, the update of the Invoice is one of the key data flows found in the integration. Invoices can be updated for a number reasons, including but not limited to: when a refund is approved, when a shipment is created and any other event that would cause a state transition.

Shipment Flows

Let’s remind ourselves of the shipment entities, (note we have included the relationship to the Invoice as it’s such a key relationship in the context of Shipments):

Shipment Entities

ProviderConsumerUse CaseDescription
MPCIShipment CreatedWhen a Seller dispatches all or part of an order a Shipment will be created in MP, this data will need to flow to the CI
MPCIShipment UpdatedShipments do have a status, which can be updated throughout the lifecycle of the Shipment, these update events should flow through to the CI.

Seller Flows

Let’s remind ourselves of the seller entities:

Seller Entities

ProviderConsumerUse CaseDescription
CIMPSeller CreatedSellers can be created via the Operator API, there may be scenarios where this is employed (e.g. if you have a Seller Sign Up page).
CIMPSeller UpdatedUpdates to existing Sellers may propagate from the CI to MP, e.g. you may want to auto offline a seller if they have breached you marketplaces terms and conditions
MPCISeller CreatedWhere new Sellers are created an on-boarded using the MP Operator Portal, you may want to propagate this date through to your CI.
MPCISeller UpdatedWhere existing Sellers are updated using the MP Operator Portal, you may want to propagate this date through to your CI.
CIMPRemittances UpdatedBefore Sellers can be paid, the operator needs to approve their release. You can automate this release process via API in which case the flow of remittance release data is from the CI to MP.
MPCIRemittances CreatedRemittances are created when certain conditions are met by a seller for their invoices, when these conditions are met, the remittance is created, but not yet released, in this case you may still want the unreleased remittance data flow from MP to the CI.
MPCIRemittances UpdatedThe remittance release process can originate in MP, in which case you may require that data to flow from MP to the CI.

Categorization Flows

Let’s remind ourselves of the categorization entities:

Categorization Entities

ProviderConsumerUse CaseDescription
MPCITaxon CreatedNew Taxons (categories) created in MP may need to flow through to the CI
MPCITaxons UpdatedExisting Taxons (categories) updated in MP may need to flow through to the CI
MPCIPrototype CreatedNew Prototypes created in MP may need to flow through to the CI
MPCIPrototype UpdatedExisting Prototypes updated in MP may need to flow through to the CI
MPCIOption Type CreatedNew Option Types created in MP may need to flow through to the CI
MPCIOption Type UpdatedExisting Option Types updated in MP may need to flow through to the CI
MPCIOption Value CreatedNew Option Values created in MP may need to flow through to the CI
MPCIOption Value UpdatedExisting Option Values updated in MP may need to flow through to the CI
CIMPTaxon CreatedNew Categories created in the eCommerce platform may need to be created in MP
CIMPTaxons UpdatedExisting Categories updated in the eCommerce platform may need to be updated in MP
CIMPPrototype CreatedNew Prototypes created in the eCommerce platform may need to be created in MP
CIMPPrototype UpdatedExisting Prototypes updated in the eCommerce platform may need to be created in MP
CIMPOption Type CreatedNew Option Types created in the eCommerce platform may need to be created in MP
CIMPOption Type UpdatedExisting Option Types updated in the eCommerce platform may need to be created in MP
CIMPOption Value CreatedNew Option Values created in the eCommerce platform may need to be created in MP
CIMPOption Value UpdatedExisting Option Values updated in the eCommerce platform may need to be created in MP

What’s Next

Having identified the most common data flows for a Connected Integration, in the next section we look at what methods you can use to move that data.