CI/CD Configuration

This guide provides instructions for setting up and using the CI/CD pipelines for the Bluefly LLM ecosystem.

Overview

The Bluefly LLM ecosystem uses GitLab CI/CD for continuous integration and deployment. The pipelines automate building, testing, and deploying the various components.

Pipeline Architecture

The CI/CD pipeline consists of several stages:

1. Build: Compile code and build artifacts
2. Test: Run various test suites
3. Document: Generate and publish documentation
4. Package: Create Docker images and packages
5. Deploy: Deploy to various environments

GitLab CI/CD Configuration

Project Structure

Each project in the ecosystem includes a .gitlab-ci.yml file that defines its pipeline. Most projects include the standard components from BFCIComponents:

include:
  - component: 'gitlab://bluefly/bfcicomponents/components/nodejs-base.yml'
    inputs:
      node_version: "18"
      project_type: "app"
  - component: 'gitlab://bluefly/bfcicomponents/components/nodejs-test.yml'
  - component: 'gitlab://bluefly/bfcicomponents/components/nodejs-build.yml'
  - component: 'gitlab://bluefly/bfcicomponents/components/documentation.yml'

Environment Variables

The following CI/CD variables should be set in your GitLab project:

Variable	Description	Example
NPM_TOKEN	Token for accessing private npm packages	abc123...
DOCKER_REGISTRY_PASSWORD	Password for Docker registry	xyz789...
DEPLOY_TOKEN	Token for deployment	def456...
DOCS_TRIGGER_TOKEN	Token for documentation pipeline	ghi789...

Pipeline Stages

Build Stage

The build stage compiles TypeScript, bundles assets, and prepares the application for deployment:

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
      - build/

Test Stage

The test stage runs unit, integration, and other tests:

unit-tests:
  stage: test
  script:
    - npm run test
  coverage: /All files[^|]*\|\s*(\d+.\d+)/

integration-tests:
  stage: test
  script:
    - npm run test:integration

Document Stage

The document stage generates and publishes documentation:

documentation:
  stage: document
  script:
    - npm run docs:generate
  artifacts:
    paths:
      - docs/

Package Stage

The package stage creates Docker images and other distribution packages:

docker-image:
  stage: package
  script:
    - docker build -t registry.gitlab.com/bluefly/project-name:$CI_COMMIT_SHA .
    - docker push registry.gitlab.com/bluefly/project-name:$CI_COMMIT_SHA

Deploy Stage

The deploy stage deploys to development, staging, and production environments:

deploy-staging:
  stage: deploy
  environment: staging
  script:
    - ./scripts/deploy.sh staging
  only:
    - develop

deploy-production:
  stage: deploy
  environment: production
  script:
    - ./scripts/deploy.sh production
  only:
    - main
  when: manual

Custom CI/CD Scripts

Each project typically includes the following CI/CD-related scripts:

prepare-ci.sh

This script prepares the repository for CI/CD pipeline execution:

#!/bin/bash
# Replace local dependencies with published versions
sed -i 's|"file:../BFCIComponents"|"@bluefly/ci-components@latest"|g' package.json
npm install

deploy.sh

This script handles deployment to different environments:

#!/bin/bash
ENVIRONMENT=$1

if [ "$ENVIRONMENT" == "production" ]; then
  # Production deployment steps
  echo "Deploying to production..."
elif [ "$ENVIRONMENT" == "staging" ]; then
  # Staging deployment steps
  echo "Deploying to staging..."
else
  echo "Unknown environment: $ENVIRONMENT"
  exit 1
fi

Setting Up a New Project

To set up CI/CD for a new project in the ecosystem:

1. Create a .gitlab-ci.yml file that includes the appropriate components from BFCIComponents
2. Add any project-specific jobs or scripts
3. Configure the required CI/CD variables in GitLab
4. Create the necessary deployment scripts
5. Set up environment-specific configuration

Troubleshooting

Common Issues

Pipeline Failure in Build Stage

• Check for TypeScript errors
• Verify that all dependencies are available
• Ensure the correct Node.js version is used

Test Failures

• Check test logs for specific errors
• Run tests locally to reproduce the issue
• Verify that tests are not dependent on local environment

Deployment Failures

• Check connectivity to deployment environment
• Verify that all required tokens and secrets are configured
• Review deployment logs for specific errors

Getting Help

If you encounter persistent issues with CI/CD pipelines:

1. Check GitLab CI/CD documentation
2. Review BFCIComponents documentation
3. Contact the Bluefly DevOps team