When onboarding your data, you can configure the dataset's delivery schedule. Crux will determine the supplier-declared schedule based on your data's file patterns and timestamps. During the data product onboarding, you will have an opportunity to accept or modify the dataset schedule.
Detecting the supplier's schedule
Crux uses file patterns of the data available at the source and files' last-modified timestamps to identify the supplier's update schedule (e.g. when the supplier actually updates data).
Configuring schedule
Crux uses 5-character Linux cron expressions.
Examples
| Deliver the dataset at the top of every hour of every day |
| Deliver the dataset every 30 minutes |
| Deliver the dataset at 8, 9, and 10 o'clock every day |
| Deliver the dataset at 8:00, 8:30, 9:00, 9:30, and 10 o'clock every day |
| Deliver the dataset on the hour nine-to-five weekdays |
Other considerations
Scheduled data extraction processing may fail due to one of the following:
Incorrect or missing configuration. Review the configuration settings for the scheduled data extraction. Ensure that the extraction settings, such as the data source location, extraction frequency, and any associated filters or parameters, are properly configured. Incorrect or missing configuration settings can prevent the extraction from executing successfully.
Connectivity or access issues. Verify that the connection is successfully created and there are no errors connecting to the data source. The scheduled data extraction may fail if there are network connectivity issues or access restrictions.
The data source is unavailable. If the data source is temporarily unavailable or experiencing downtime during the scheduled data extraction, it can result in data downtime or other failures. Check the availability of the data source and ensure it is accessible and operational during the scheduled extraction time.
Timezone, daylight savings, or scheduling issues. Verify the timezone settings and scheduling parameters for the extraction. Discrepancies or misconfigurations related to the timezone, time offsets, or daylight saving time can impact the execution of the scheduled data extraction. Ensure that the scheduling settings align correctly with the intended extraction time.
By considering these potential causes, you can investigate and address the factors contributing to the failure of a scheduled extraction process. Analyzing the specific error messages or logs related to the failed extraction can also provide valuable insights into the root cause of the issue.