Skip to main content

Build Scripts Guidelines

This document provides conventions and best practices for shell scripts in the ProvLabs documentation build system.

Script Structure​

File Organization​

  • All build scripts are located in the scripts/ directory
  • Use executable permissions (chmod +x) for all shell scripts
  • Use .sh extension for all shell scripts

Script Headers​

#!/bin/bash

# Brief description of what this script does
# Include usage examples if the script takes parameters

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

Error Handling​

Exit Codes​

  • Use || exit for commands that must succeed
  • Use set -e at the top of scripts to exit on any error
  • Provide meaningful error messages before exiting

Directory Operations​

# Good: Safe directory operations
pushd "$SCRIPT_DIR/.." || exit
# ... operations ...
popd || exit

# Good: Check if directories exist before operations
if [[ ! -d "$TARGET_DIR" ]]; then
  mkdir -p "$TARGET_DIR"
fi

OpenAPI Generation Patterns​

Protocol Buffer Processing​

  • Use buf generate for generating OpenAPI specs from proto files
  • Apply consistent filtering for query and service proto files only
  • Clean up temporary directories after generation

File Processing Patterns​

# Standard pattern for processing proto directories
proto_dirs=$(find ./path -path -prune -o -name '*.proto' -print0 | xargs -0 -n1 dirname | sort | uniq)
for dir in $proto_dirs; do
  query_file=$(find "${dir}" -maxdepth 1 \( -name 'query.proto' -o -name 'service.proto' \))
  if [[ ! -z "$query_file" ]]; then
    buf generate --template ../../template.yaml -o ../ "$query_file"
  fi
done

OpenAPI Manipulation​

  • Use jq for JSON manipulation and schema removal
  • Use redocly for linting, joining, and splitting OpenAPI specifications
  • Apply consistent schema cleanup with removeBaseQuerySchemas function

File Management​

Cleanup Operations​

# Safe removal patterns
rm -rf "$OPENAPI_DIR"
mkdir -p "$OPENAPI_DIR"

# Conditional cleanup
if [[ -d "$TEMP_DIR" ]]; then
  rm -r "$TEMP_DIR"
fi

File Copying​

# Recursive copying with proper error handling
cp -R source_dir/* destination_dir/
mv source_file destination_file

Submodule Operations​

Git Submodule Patterns​

  • Work within submodule directories using pushd/popd
  • Generate files outside of submodule directories to avoid git issues
  • Clean up generated files within submodules after moving them

Dependencies​

Required Tools​

Scripts assume the following tools are available:

  • buf - Protocol Buffer tooling
  • jq - JSON processing
  • redocly - OpenAPI tooling
  • Standard Unix tools (find, xargs, sed, mv, cp, rm)

Tool Verification​

# Check for required tools at script start
command -v buf >/dev/null 2>&1 || { echo "buf is required but not installed"; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "jq is required but not installed"; exit 1; }

Schema Management​

Base Schema References​

  • Remove duplicate schemas to avoid conflicts when joining specs
  • Update references to point to centralized base schema file
  • Use consistent sed patterns for reference replacement

Schema Cleanup Function​

function removeBaseQuerySchemas() {
  SVC_FILE=$1

  jq "del( \
        .components.schemas.\"schema.name.1\", \
        .components.schemas.\"schema.name.2\" \
        )" "$SVC_FILE" > "$SVC_FILE.tmp"

  mv "$SVC_FILE.tmp" "$SVC_FILE"

  # Update references
  sed -i '' -e "s|old-reference|new-reference|g" "$SVC_FILE"
}

Integration Points​

npm Script Integration​

  • Scripts are called from package.json scripts (e.g., prepare script)
  • Ensure scripts can be run from repository root
  • Scripts should be idempotent - safe to run multiple times

Build Pipeline​

  • protoc-openapi.sh generates all OpenAPI files
  • split-openapi.sh is called automatically by protoc script
  • tag-operations.sh and post-provenance.sh handle post-processing
  • Final output should be ready for Docusaurus build

Performance Considerations​

Parallel Processing​

  • Use find with xargs for efficient file processing
  • Process multiple proto files in batches when possible
  • Clean up intermediate files promptly to save disk space

Caching Strategy​

  • Generated files in openapi/ directory are not committed
  • Intermediate files should be cleaned up after use
  • Consider timestamp-based regeneration for development efficiency