Managing Disk Space, Scheduling, Progress, and Timeouts in OCI Batch
Learn how to manage OCI Batch jobs by configuring disk space requirements, understanding service-defined timeouts, handling insufficient resources, and monitoring job progress.
Configuring Minimum Disk Space Requirements
A task profile can specify the minimum fleet node boot-volume size required for scheduling. This requirement doesn't guarantee the same capacity in the task container file system.
Before You Begin
Create a batch context with a fleet that meets the task's resource requirements. For fleet boot-volume sizing and eligibility rules, see Matching Tasks to Fleets.
Configuring Minimum Disk Space for a Task
Specify the minimum disk size when you create a task profile, then use that profile for the task. For the procedure, see Creating a Task Profile.
To change the resource requirements, create a new task profile.
Fleet Boot-Volume Sizing
For explicit and automatically derived fleet boot-volume sizes and applicable minimums, see Matching Tasks to Fleets.
Benefits
Specifying a minimum disk size makes the task's scheduling requirement explicit and limits eligibility to fleets whose node boot volumes meet that requirement.
Understanding Insufficient Resource Handling
OCI Batch improves scheduling behavior when no configured fleet can satisfy a job's resource requirements.
When OCI Batch evaluates a job, it compares the resource requirements defined in the task profile with the available resources across all configured fleets. If no fleet satisfies the requirements, OCI Batch fails the task during scheduling instead of leaving it in a waiting state.
This behavior helps administrators identify configuration issues immediately and take corrective action.
How Resource Validation Works
During scheduling, OCI Batch verifies that at least one configured fleet meets all task resource requirements.
For example, if a task profile requires 500 GB of disk space and the largest configured fleet provides 200 GB, OCI Batch accepts the job request and assigns a job ID. During scheduling, OCI Batch determines that no eligible fleet satisfies the minimum disk space requirement and fails the task with a scheduling error.
The task details include the reason for the failure. You don't need to inspect logs to determine why the task wasn't scheduled.
Resolving Resource Mismatches
If a task fails because no eligible fleet satisfies its resource requirements, you can take one or more of the following actions:
- Create or select a Batch context that includes a fleet with sufficient capacity.
- Update the task profile to reduce the resource requirements.
- Modify the fleet configuration to provide additional resources.
- Resubmit the job after resolving the resource mismatch.
Benefits
Immediate scheduling failures provide the following benefits:
- Identify resource configuration issues during scheduling.
- Reduce troubleshooting time by providing clear failure reasons.
- Prevent tasks from remaining in an indefinite waiting state.
- Help administrators resolve capacity issues before resubmitting workloads.
Monitoring Job Progress
OCI Batch provides job progress information that helps you monitor execution across all tasks in a job.
The job progress view displays the overall execution status, including the number of succeeded, failed, canceled, and running tasks. OCI Batch also calculates the overall completion percentage for the job.
Job progress is the percentage of all tasks in the job that have reached a terminal state:
Progress (%) = (Number of tasks in a terminal state / Total number of tasks in the job) × 100
Tasks that have succeeded, failed, or been canceled count toward progress once they reach a terminal state. Progress measures completion, not the success rate. A job can reach 100 percent progress even if some tasks failed or were canceled.
You can use this information to monitor long-running jobs and identify potential issues during execution.
Viewing Job Progress
You can monitor job progress from the following views:
- Job progress percentage
- Task summary counts
- Job details page
The job progress view is updated as task execution progresses.
Interpreting Job Progress
The job progress view displays the current status of all tasks associated with a job. For example, a job with 40 compute tasks might show the following status:
- 20 tasks succeeded
- No tasks failed
- No tasks were canceled
- 20 tasks are still running
Based on this information, OCI Batch reports the job as 50 percent complete.
Monitoring Job Execution
Review job progress throughout execution to verify that the workload is progressing as expected.
Changes in the reported task counts can help you identify potential issues. For example:
- If the completion percentage doesn't increase over time, the workload might no longer be making progress.
- If the number of failed tasks begins to increase, review the affected tasks to determine the cause before the job completes.
Benefits
Monitoring job progress provides the following benefits:
- Tracks overall job completion during execution.
- Provides visibility into the status of individual tasks.
- Helps identify stalled or failed workloads earlier.
- Improves operational monitoring for long-running jobs.
- Supports informed decisions when troubleshooting or reporting workload status.
Understanding Job and Task Timeouts
Batch enforces service-defined timeout limits for jobs and tasks. You can't configure these limits when you create or update a job.
Job Inactivity Timeout
A job times out after 28 days without lifecycle activity or user updates. The timeout interval is measured from the job's last update. A long-running job that continues to make progress doesn't time out solely because it was created more than 28 days ago.
Service-Defined Timeout Limits
- Jobs: 28 days without activity, measured from the last job update.
- Tasks: 14 days of task execution.
Timeout Lifecycle Transitions
- If a task exceeds its 14-day execution limit, the job transitions from In Progress to Failed.
- If a job reaches its inactivity timeout while tasks are running, the job transitions from In Progress to Canceling and then Failed. Its
lifecycleDetailsvalue isTIMEOUT.
For more information about job lifecycle states, see Batch Job Lifecycle States.
Reviewing Job Results
After a timeout, review the job's lifecycle details and the final status of each task to determine which tasks completed before the job stopped.
Benefits
Service-defined timeouts provide the following benefits:
- Stops inactive jobs and tasks that exceed the execution limit.
- Reduces unnecessary resource consumption.
- Helps control infrastructure costs.
- Simplifies identification of stalled or unhealthy workloads.
- Provides clear lifecycle information for troubleshooting timeout events.
Example Scenarios
Minimum Disk Space
For an example of matching a task's minimum disk requirement to fleet node boot-volume sizes, see Matching Tasks to Fleets.
Job Progress
A job contains 40 compute tasks. During execution, the job progress view reports the following status:
- 30 tasks succeeded
- No tasks failed
- No tasks were canceled
- 10 tasks are still running
Based on the current task status, OCI Batch reports the job as 75 percent complete. The job progress view also displays the overall completion percentage, task summary counts, and detailed task status.
Administrators can use this information to monitor job execution, verify that the workload is progressing as expected, and identify potential issues. For example, if the completion percentage stops increasing or the number of failed tasks begins to increase, investigate the affected tasks to determine the cause before the job completes.
Job Timeout
A job has running tasks but has had no lifecycle activity or user updates for 28 days. The job reaches its service-defined inactivity timeout.
OCI Batch cancels the running work and marks the job as failed. The job lifecycle transitions through the following states:
- In Progress
- Canceling
- Failed
The job's lifecycleDetails value is TIMEOUT. Review the lifecycle history and task results to determine when the job timed out and which tasks completed before it stopped.
Insufficient Resources
A task profile specifies a minimum disk space requirement of 500 GB, but the largest configured fleet provides 250 GB of disk space.
OCI Batch accepts the job request and assigns a job ID. During scheduling, OCI Batch evaluates the available fleet capacity and determines that no configured fleet satisfies the minimum disk space requirement.
The task fails immediately with a scheduling error that explains that no eligible fleet can satisfy the requested resources. Review the task details to identify the resource mismatch. Before resubmitting the job, create a task profile with compatible resource requirements or create a fleet that satisfies the task requirements.