A data product is the core "unit of exchange" in Crux, representing a package that delivers data from an external source to a target destination. It is a collection of one or more datasets (and their associated workflows), all sourced from the same data provider, covering a specific subject area under a single licensing agreement. Data products are at the top of the Crux domain model, and they include everything needed to manage and deliver data.
After data is extracted (ingested from the source) and processed (normalized and standardized), it becomes a data product. A data product may consist of multiple datasets, each with its own delivery schedule and pipeline. Data products can be sourced from a single data source or transformed from the output of other data products through Crux Wrangle, a service available to Sphere by Crux - Managed Service customers.
Components of a data product
Every Crux data product is composed of the following core components:
Source Connection: Establishes the link to the data provider, where the raw data originates.
Destination Connection: Defines where the processed data will be delivered, such as a data lake, data warehouse, or other storage solution.
Dataset(s): The fundamental structure within a data product that organizes data into manageable units. Each dataset can have its own processing schedule and data pipeline.
Delivery: The process responsible for fetching the data from the source at regular intervals, ensuring that the dataset is always up-to-date.
Summary
A data product in Crux is designed to manage and deliver external data efficiently. It consists of one or more datasets organized into tables and table schemas to support flexible data processing. With its components—source and destination connections, datasets, and deliveries—a data product ensures that data is regularly updated, normalized, and model-ready.
Learn more
Explore related concepts of the Crux domain model to build successful data products.