Overview
After you have onboarded a data product using Crux Studio, you may run into issues during the life of the active data product that need further exploration. Viewing logs in the app is not supported at this time. However, you may retrieve service logs using cruxctl to better understand why certain failures occur whether it's during ingestion, processing, or delivery of your data.
Viewing failures in the Health Dashboard
When a failure occurs during any point in the lifecycle of data flow from source to destination, Crux does a best guess of what occurred.
However, there are many permutations to what can go wrong with external data. The different services that contain logs are as follows:
PDK service: this platform service handles pipeline DAG runs for ingesting and processing data. If a failure occurs in either the Retrieved by Crux or Processed by Crux steps, you may explore the PDK logs with the
cruxctl dataset pdk-logscommand.Dispatch service: this platform service handles the dispatch of Crux-processed data to a target destination. If a failure occurs on the Delivered to destination step, you may explore the Dispatch logs with the
cruxctl dataset dispatch-logscommand.
Logs for ingestion or processing failures
Run the following command to learn about PDK logs:
cruxctl dataset pdk-logs --help
This will print out a short tutorial for using the cruxctl dataset pdk-logs command.
In order to run logs for the PDK service to investigate failures at ingestion or processing, you will need the following parameters:
Dataset ID: This is a unique identifier for the dataset you are interested in and is available in the Health Dashboard when viewing dataset details. The ID has a format of
Ds123456.Delivery ID: This is a unique identifier for the data delivery attempt and is available in the Health Dashboard when viewing dataset details. You may hover on the delivery ID and quickly copy it to your clipboard.
Logs for delivery failures
Run the following command to learn about Dispatch logs:
cruxctl dataset dispatch-logs --help
This will print out a short tutorial for using the cruxctl dataset dispatch-logs command.
In order to run logs for the Dispatch service to investigate failures in delivering to destinations, you will need the following parameters:
Dataset ID: This is a unique identifier for the dataset you are interested in and is available in the Health Dashboard when viewing dataset details. The ID has a format of
Ds123456.Export ID: