Module: Train and Deploy Models

Lesson: Running a Script as a Job

Introduction: From Local Notebooks to Production Infrastructure

When you are in the early stages of data science or machine learning development, you likely spend most of your time in an interactive environment like a Jupyter Notebook. You run code cell-by-cell, visualize data, and tweak hyperparameters in real-time. While this is excellent for exploration and rapid prototyping, it is fundamentally unsuitable for the actual training of models that need to run for hours, days, or across distributed clusters of hardware.

Running a script as a "job" is the process of decoupling your code from your local machine and submitting it to a dedicated compute environment. This shift is critical because local machines are often unreliable; if your laptop lid closes, the internet drops, or the power goes out, your training run dies. By packaging your code into a script and submitting it as a managed job, you ensure that the task runs to completion, regardless of the state of your local workstation.

This lesson explores the transition from interactive development to job-based execution. We will look at how to structure your code for automation, how to handle configuration, how to monitor remote processes, and how to adopt professional standards that make your workflows reproducible and scalable.

Understanding the "Job" Paradigm

In a production environment, a "job" is an encapsulated unit of work. It consists of three primary components: the code (your script), the environment (the libraries and dependencies required), and the compute resources (the CPU, GPU, and memory allocated to the task).

When you run a script as a job, you are essentially telling an orchestrator—whether that is a simple Linux cron job, a Kubernetes cluster, or a cloud-based service like AWS Batch or Google Vertex AI—to perform a task in isolation. This isolation is vital. It prevents your training run from interfering with other processes on your machine and ensures that your environment is consistent every time the code runs.

Callout: The "It Works on My Machine" Problem One of the most common hurdles in machine learning is the discrepancy between a developer’s local environment and the production server. By containerizing your code and running it as a job, you force the inclusion of a dependency file (like requirements.txt or a Dockerfile). This ensures that the exact same library versions are used during training, validation, and deployment, effectively eliminating the "it works on my machine" phenomenon.

Step 1: Refactoring Code for Non-Interactive Execution

The first step in moving to job-based execution is refactoring your code. Notebooks are linear and often contain fragmented state; scripts, by contrast, must be modular and self-contained.

Use Argument Parsing

You should never hard-code hyperparameters or file paths inside your script. Instead, use a library like argparse to allow external control over the job's behavior. This allows you to run the same script multiple times with different settings without modifying the source code.

import argparse

def train_model(data_path, learning_rate, epochs):
    print(f"Loading data from {data_path}")
    print(f"Training with LR={learning_rate} for {epochs} epochs")
    # Training logic goes here
    pass

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Model Training Job")
    parser.add_argument("--data_path", type=str, required=True, help="Path to input data")
    parser.add_argument("--lr", type=float, default=0.001, help="Learning rate")
    parser.add_argument("--epochs", type=int, default=10, help="Number of epochs")
    
    args = parser.parse_args()
    train_model(args.data_path, args.lr, args.epochs)

By structuring your code this way, you can trigger your job from a command line or a CI/CD pipeline using: python train.py --data_path ./data.csv --lr 0.01 --epochs 50.

Logging Instead of Printing

In an interactive notebook, you see print statements immediately. In a remote job, you need a persistent record. Replace standard print() statements with the Python logging module. This allows you to capture timestamps, severity levels (INFO, WARNING, ERROR), and route the output to a file that can be inspected after the job finishes.

Step 2: Managing Dependencies and Environments

When your script executes on a remote server, that server needs all the libraries you used. You have two main ways to handle this:

Virtual Environments: Using venv or conda to create a local directory containing the project-specific libraries.
Containers (Docker): Packaging the OS, the Python runtime, and all libraries into a single immutable image.

For professional machine learning workflows, containers are the industry standard. They provide the highest level of predictability.

Creating a Dockerfile

A Dockerfile acts as a set of instructions for building the environment. Here is a simple example:

# Use an official lightweight Python image
FROM python:3.9-slim

# Set the working directory
WORKDIR /app

# Copy dependency file
COPY requirements.txt .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the source code
COPY . .

# Run the training script
ENTRYPOINT ["python", "train.py"]

Note: Always use specific versions for your dependencies (e.g., scikit-learn==1.2.2 instead of just scikit-learn). This prevents your jobs from breaking unexpectedly when a library releases a new, potentially incompatible update.

Step 3: Submitting the Job

Once your code is modular and containerized, you need to execute it. Depending on your organization's infrastructure, this might be handled in several ways.

Using a Local Job Runner (The Simple Case)

If you are just starting out, you can simulate a job environment using a tool like nohup on a Linux server. This allows a process to keep running even after you disconnect your SSH session.

nohup python train.py --data_path ./data.csv > training.log 2>&1 &

nohup: Prevents the process from dying when you log out.
> training.log: Redirects standard output to a log file.
2>&1: Redirects error messages to the same log file.
&: Puts the process in the background.

Using Cloud Job Services

Most modern machine learning is done on cloud platforms. These services provide "Job Submission" APIs. Instead of managing a server yourself, you provide a reference to your Docker image and the command to run.

Upload the image to a registry (like Docker Hub or ECR).
Define the compute requirements (e.g., 4 vCPUs, 16GB RAM, 1 NVIDIA T4 GPU).
Submit the job via a command-line tool or SDK.

Best Practices for Robust Training Jobs

Running jobs is not just about execution; it is about visibility and reproducibility. If a job fails, you need to know why. If a job succeeds, you need to know what result it produced.

1. Implement Checkpointing

Long-running training jobs are susceptible to hardware failures or preemption (if using spot instances). Implement checkpointing by saving your model state (weights, optimizer status) to a persistent storage location (like an S3 bucket or a shared network drive) every few epochs. If the job crashes, you can resume from the last checkpoint rather than starting from scratch.

2. Externalize Configuration

Keep your configurations (hyperparameters, dataset paths, training duration) in a YAML or JSON file. Your script should read this file at startup. This allows you to version-control your experiments by committing these config files to your Git repository, creating a perfect audit trail of what was tested.

3. Use Metrics Tracking

Do not rely on log files alone. Use tools like MLflow, Weights & Biases, or TensorBoard to log metrics (accuracy, loss, F1-score) in real-time. These tools provide a graphical interface to compare different job runs, which is essential for hyperparameter tuning.

4. Handle Cleanup

Ensure your job cleans up after itself. If your job creates temporary files, ensure they are deleted upon completion or failure. If you are spinning up cloud resources, ensure they are terminated once the job finishes to avoid unnecessary costs.

Comparison of Execution Methods

Feature	Local Notebook	Script (SSH/Nohup)	Managed Cloud Job
Persistence	Low (stops if connection lost)	Medium (resilient to disconnect)	High (durable, auto-restarts)
Scalability	None	Manual	Automatic/Orchestrated
Reproducibility	Poor	Medium	Excellent
Monitoring	Interactive	Log files	Real-time Dashboards
Cost	Fixed	Fixed (always on)	Pay-per-use

Common Mistakes and How to Avoid Them

Even experienced engineers fall into common traps when moving from notebooks to jobs. Awareness of these pitfalls will save you significant debugging time.

The "Hard-Coded Path" Trap

Mistake: Writing df = pd.read_csv("/Users/username/project/data.csv") inside your code. Fix: Always use relative paths or environment variables. Pass the input data path as an argument to the script. This ensures that the code runs on your local machine, the build server, and the production cluster without modification.

The "Missing Dependency" Trap

Mistake: Installing a new package via !pip install inside a notebook and forgetting to add it to your requirements.txt. Fix: Always maintain your requirements.txt as you go. If you use a new library, add it to the file immediately. Run a fresh environment build locally before pushing your code to ensure the file is complete.

The "Resource Exhaustion" Trap

Mistake: Submitting a job that requires 32GB of RAM to a machine with only 8GB of RAM. Fix: Monitor your resource usage during local testing. If you don't know the requirements, start with a larger instance than you think you need, then scale down based on the actual usage metrics observed in your first few runs.

The "Silent Failure" Trap

Mistake: Writing code that catches all exceptions and ignores them. Fix: Never use try...except: pass. If a job fails, the system should know it failed. Ensure your script exits with a non-zero status code (sys.exit(1)) if an unrecoverable error occurs. This signals the orchestrator that the job has failed, allowing it to trigger alerts or retry logic.

Step-by-Step: From Local Script to Managed Job

To solidify your understanding, follow this workflow to move a training script to a managed environment.

Modularize: Move your core logic into a function (e.g., def run_training()).
Externalize: Add argparse to handle inputs like data_path and batch_size.
Containerize: Create a Dockerfile that includes all necessary system and Python libraries.
Test Locally: Run the container locally using docker run to ensure it executes the same way as your script.
Push: Push your container image to a registry.
Submit: Use your organization’s job submission tool to trigger the container, passing the appropriate arguments.
Monitor: Check the logs or your experiment tracking dashboard (like MLflow) to verify the job is progressing.

Warning: Secrets Management Never hard-code API keys, database credentials, or tokens in your scripts or Dockerfiles. These will be stored in plain text in your version control system. Instead, use environment variables or a secret management service (like HashiCorp Vault or AWS Secrets Manager) to inject these credentials into the job at runtime.

Designing for Failure: The "Idempotent" Concept

A key concept in job-based execution is idempotency. An idempotent job is one that can be run multiple times with the same input, and the result will always be the same, without unintended side effects.

For example, if your training job saves the model file as model_v1.pkl every time it runs, it might overwrite a previous version. An idempotent approach would be to save the model using a timestamp or a unique run ID (e.g., model_20231027_1400.pkl). This ensures that if you accidentally trigger the job twice, you don't lose the results of the first run.

Furthermore, if your job writes data to a database, ensure it checks if the data already exists before inserting. This makes your training pipelines much more resilient, as you can safely retry failed jobs without worrying about duplicating data or corrupting your state.

Scaling Up: Distributed Training

Once you master running a single script as a job, you will eventually reach a point where the model is too large or the dataset too vast for a single machine. This is where distributed training comes into play.

Distributed training involves breaking your job into multiple smaller tasks that run in parallel on different machines. While this is an advanced topic, the principle remains the same: you are still running a script as a job. The orchestrator simply triggers that same script across a cluster of nodes, providing each node with a different subset of the data or a different part of the model.

If you are using frameworks like PyTorch or TensorFlow, they have built-in support for distributed training (e.g., DistributedDataParallel). You write your script to handle the distribution, and your job orchestrator handles the provisioning of the hardware.

Advanced Monitoring: Beyond Logs

While text logs are essential, they are not always enough for complex machine learning jobs. Consider implementing these three levels of monitoring:

System Metrics: CPU usage, GPU temperature, memory utilization, and network throughput. Use tools like Prometheus and Grafana for this. It helps identify if your job is bottlenecked by the hardware (e.g., if the GPU is sitting idle, you might have a data loading bottleneck).
Process Metrics: Training loss, validation accuracy, gradient norms. Use experiment trackers like Weights & Biases or MLflow. This tells you if the model is actually learning.
Business/Data Metrics: Distribution of predicted values, drift detection, or specific evaluation metrics on a hold-out test set. This tells you if the model is performing well in the real world.

The Role of CI/CD in Machine Learning (MLOps)

In a mature environment, running a job manually becomes a chore. This is where Continuous Integration and Continuous Deployment (CI/CD) come in.

Every time you commit code to your repository, a CI pipeline should:

Run unit tests on your code to ensure no bugs were introduced.
Build a new Docker image containing the updated code.
Push the image to your registry.

Once the image is built, a CD pipeline can automatically deploy or trigger a "smoke test" job to ensure the new model trains correctly on a small subset of data. This automation ensures that your training infrastructure is always ready and that you are not manually building and pushing images every time you make a change.

Troubleshooting Common Job Failures

Even with the best practices, jobs will occasionally fail. Here is a quick reference guide for diagnosing common issues:

Job exits with code 137: This usually indicates an "Out of Memory" (OOM) error. The OS killed your process because it requested more memory than available. Increase the memory allocation for the job.
Job hangs indefinitely: This often happens due to deadlocks in multi-threaded code or waiting for a network resource that never responds. Set timeouts for all network operations.
Job fails immediately with "Module Not Found": Your Docker image is missing a dependency. Check your requirements.txt and ensure it is being copied into the Docker container correctly.
Job runs but produces no output: Ensure you are writing your artifacts to a persistent volume or a cloud storage bucket. If you write to the container's local filesystem, those files will disappear when the container terminates.

Key Takeaways

Decouple and Automate: Move away from notebooks for long-running processes. Refactor your code into modular scripts that take arguments, allowing for repeatable, automated execution.
Containerization is Essential: Use Docker to package your environment. This creates a portable, consistent runtime that eliminates environment-related bugs across different machines.
Prioritize Observability: Replace print statements with robust logging. Integrate experiment tracking tools to monitor metrics in real-time, providing visibility into your model's performance.
Design for Resilience: Implement checkpointing and ensure your jobs are idempotent. This allows your workflows to recover from failures automatically without data loss or corruption.
Externalize Configuration: Never hard-code parameters. Use configuration files to manage hyperparameters and environment settings, enabling better version control and experimental tracking.
Secure Your Work: Use secret management services for credentials and tokens. Never commit sensitive information to your code repository.
Standardize the Workflow: As you scale, adopt CI/CD practices to automate the building and testing of your training jobs, reducing manual effort and increasing the speed of your development cycle.

By moving from interactive exploration to structured, job-based training, you transform from a researcher into a machine learning engineer. This transition is the single most important step in building production-ready systems that can reliably deliver value. Start small, containerize your first script, and you will quickly see how much more stable and efficient your development process becomes.

Running a Script as a Job