Create a Managed Inference Job (web interface)

Create a Managed Inference Job in the CosmicAC web interface, then call your model.

Create a Managed Inference Job in the CosmicAC web interface. You set the basics, select a model, configure the endpoint and hardware, then launch the job. The Job configuration reference describes every field.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
Access to the CosmicAC web interface.

Steps

Open the Models page

In the left navigation, click Models to open the Models page.

Open the new job form

On the Models page, click Deploy model.

CosmicAC recommends serving parameters, environment variables, and hardware for each supported model. If you don't follow these recommendations, the model can fail to deploy. See Recommended model parameters.

Enter the basics

In the Basics section, enter a Job name as a lowercase name with hyphens, choose a Location, and add Tags. To add a tag, type it and press Enter or comma.

Select the job type

In the What kind of job? section, select Managed Inference.

Select a model

In the Model to serve section, select a Model. The configuration fields depend on the model's runtime.

For a vLLM model, set the Serving configuration:

Runtime image (CUDA) — the vLLM serving image.
Data type — the numeric precision the model runs at.
Quantisation — how to compress the model weights.
Tensor parallel — how many GPUs to split the model across.
GPU memory utilisation — the fraction of GPU memory to use.
Max model length — the maximum context length.
Max concurrent sequences — the maximum requests handled at once.
Reasoning parser — the parser for reasoning output.
Video & image input — whether the model accepts multimodal input.
Root disk size — the VM root disk size in GB.

These fields come prefilled with the model master's defaults. Apply the Recommended model parameters for supported models.

For a Parakeet model (nvidia/parakeet-tdt-0.6b-v3), set the Parakeet configuration:

Chunk duration — the audio chunk length in seconds.
Chunk overlap — the overlap between chunks in seconds.
Max file size (MB) — the maximum audio upload size.

Set environment variables

In the Environment variables part of Serving configuration, review the prefilled variables and edit them if needed. To add another, click Add variable, then enter a Name and Value.

A Parakeet job has no environment variables, so skip this step.

Configure the endpoint

Still in the Model to serve section, set the Endpoint name and Replicas under Endpoint. The endpoint name must be globally unique across active inference jobs.

A Parakeet job has no replicas, so set only the Endpoint name.

Require an API key

Under API key required, keep Require Authorization header enabled. See Create an API key.

Choose the hardware

In the Hardware section, select a GPU and the GPU count, then set the CUDA / driver.

Review and create the job

In the Review & launch section, confirm the job spec is valid, then click Create job.

Open the endpoint

Wait for the job to go live, then click Open endpoint.

Call your model

Copy the endpoint URL, then send a request with your API key in the Authorization header. Use the example request shown on the endpoint as a starting point.