Secrets#

The following secrets are needed for running a fully functional pipeline using pipeline-as-code. Some of the secrets are auto-distributed in the build namespaces of all tenants. Organization level secrets will be deployed through the infra repository. Repository and application level secrets will be deployed through GitOps repository.

SAAP Managed Secrets#

sonar-creds
- Purpose: Used by sonarqube-scan pipeline task.
- Owner: SAAP admins.
- Type: Login credentials for SonarQube.
- Used for: For running SonarQube scan in pipeline.
- Lifecycle: Every time a new tenant is created, the secret gets deployed in the build namespace. SonarQube credentials are not rotated and remain the same.
- Comment: The origin of this secret is the SonarQube namespace. Secret is copied over to build namespace using an MTO template and Template Group Instance.
- Deployment Process: The SonarQube deployed on SAAP contains a secret named sonar-creds in its namespace. This secret contains the username and password for SonarQube. We use a Multi Tenant Operator Template and TemplateGroupInstance to copy this secret and distribute it the build namespaces of all tenants. The Template and TemplateGroupInstance are both named sonar-creds.
docker-reg-creds
- Purpose: Used by buildah task and the application deployment to pull the image from the nexus registry.
- Owner: SAAP admins.
- Type: Login credentials for nexus docker registry. The secret itself is of type dockerconfigjson.
- Used for: Pulling images from the nexus registry. Needs to be deployed in all namespaces of the tenant. We distribute it using a TGI.
- Lifecycle: Every time a new tenant is created, the secret gets deployed in all its namespaces.
- Deployment Process: Nexus comes shipped with SAAP. The nexus3 namespace contains a secret named docker-reg-creds. This secret contains the .dockerconfigjson file. We use a Multi Tenant Operator Template and TemplateGroupInstance to copy this secret and distribute it all namespaces of the tenants. The Template and TemplateGroupInstance are both named docker-reg-creds.
helm-reg-creds
- Purpose: Used to pull and push charts from the Nexus Helm Registry. We use it in two places for our pipeline:
  1. stakater-helm-push task
  2. ArgoCD to fetch the helm chart
- Owner: SAAP Admins.
- Used for: Pulling charts from Nexus.
- Lifecycle: Every time a new tenant is created, the secret gets deployed in the build namespace. The same secret is deployed in the rh-openshift-gitops-instance when SAAP is provisioned.
- Deployment Process: Nexus comes shipped with SAAP. The nexus3 namespace contains a secret named helm-reg-creds. This secret contains the username and password for the helm registry. We use a Multi Tenant Operator Template and TemplateGroupInstance to copy this secret and distribute it all namespaces of the tenants. The Template and TemplateGroupInstance are both named helm-reg-creds. Another TGI named helm-reg-creds-gitops deploys the secret in GitOps namespace so ArgoCD can fetch the charts.
rox-creds
- Purpose: Used by three Tekton Tasks:
  1. stakater-rox-deployment-check
  2. stakater-rox-image-check
  3. stakater-rox-image-scan
- Owner: SAAP admins
- Used for: Communicating with RHACS API to scan images and deployments
- Lifecycle: Created at the time of RHACS deployment. The secret is then copied over to build namespaces of tenants.
- Comment: Needs to be deployed in build namespace. We deploy it using TGI.
- Deployment Process: After StackRox is installed on the SAAP cluster. An API token is created and stored in the rox-creds secret in the stakater-stackrox namespaces. We then use a Template and a TemplateGroupInstance with the same name to distribute the secret in the build namespace of tenants.

Customer Managed Secrets#

ArgoCD authentication with `infra-gitops-config` Repository#

infra-gitops-creds
- Purpose: This secret is added so ArgoCD can sync the repository. You can either use an ssh key or a personal access token for this purpose.
- Owner: The owner of this secret will be customer's delivery engineer
- Location: The secret will be deployed in the rh-openshift-gitops-instance namespace.
- Used for: Use only for the purpose of syncing your infra GitOps repository with ArgoCD
- Format: Given below is the template for this secret. The secret/external secret will need to have argocd.argoproj.io/secret-type: repository label on it:
```
apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
labels:
   argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@github.com:argoproj/my-private-repository
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----
```
- Scopes: When creating personal access token for this, token should have the following scopes attached.
- Comment: This secret needs to be deployed on the cluster directly.
Note

These secrets need to go into your Infra GitOps Repository

ArgoCD authentication with `apps-gitops-config` Repository#

apps-gitops-creds

Purpose: This secret is added so ArgoCD can sync the apps-gitops-config repository.
Owner: The owner of this secret will be customer's delivery engineer
Location: The secret will be deployed in the rh-openshift-gitops-instance namespace through the infra-gitops repository
Format: Will have the same format as that of infra-gitops-creds secret
Use for: Syncing apps GitOps repository
Comment: Once you have the both the repositories bootstrapped with ArgoCD, the first thing we will need to do for our pipelines to function is to connect pipeline-as-code to our applications repository. We do this using a Repository CR. The Repository CR references a couple of secrets to connect with the application's repository in the SCM.

Deployment Process: To deploy the apps-gitops-creds, follow the below-mentioned steps:

Navigate to your infra-gitops-config repository
At the base level, your infra repository should already have a folder with cluster name. You can refer to this tutorial for defining your Infra GitOps Repository structure. Open up the relevant cluster folder.
Inside it, create a folder named gitops-repositories

Now add an external secret that has the following structure. Remember to replace the placeholder:

  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: apps-gitops-creds
    namespace: rh-openshift-gitops-instance
  spec:
    secretStoreRef:
      name: stakater-cluster-secret-store
      kind: ClusterSecretStore
  data:
  - remoteRef:
      key: git-pat-creds
      property: username
    secretKey: username
  - remoteRef:
      key: git-pat-creds
      property: password
    secretKey: password
  target:
    name: apps-gitops-creds
    template:
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
    data:
      name: apps-gitops-creds
      password: "{{ .password | toString }}"
      username: "{{ .username | toString }}"
      project: TENANT_NAME
      type: git
      url: "https://github.com/DESTINATION_ORG/apps-gitops-config.git"

Now open up Vault and open the common-secrets path. Add a secret named git-pat-creds and add two key 'password' and 'username'. Password should have Personal Access Token with that can access your apps-gitops-config repository.

Now go to the argocd-apps folder in the infra-gitops-config repo and add and ArgoCD application pointing to your gitops-repositories folder:

  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: gitops-repositories
    namespace: rh-openshift-gitops-instance
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  spec:
    destination:
      server: 'https://kubernetes.default.svc'
    source:
      path: cluster-name/gitops-repositories
      repoURL: YOUR_INFRA_REPO_URL
      targetRevision: main
      directory:
        recurse: true
    project: default
    syncPolicy:
      automated:
        prune: true
        selfHeal: true

Wait for ArgoCD to sync your changes

Organization Level Secrets#

git-pat-creds

Purpose: Used for three reasons:
1. In the Repository CR so pipeline-as-code can talk to the repository
2. In create-environment task to get commit hashes
3. In TronadorConfig to allow Tronador to access the application repository
Owner: The owner of this secret will be customer's delivery engineer.
Location: This secret will be deployed in build namespace of all tenants, the namespaces created by Tronador

Deployment Process: To deploy the git-pat-creds, follow the below-mentioned steps:

Navigate to your infra-gitops-config repository
At the base level, your infra repository should already have a folder with cluster name. Open up the tenant-operator-config and create a folder named templates if it is not already there.

Now add a template with the following structure. Remember to replace the placeholders:

  apiVersion: tenantoperator.stakater.com/v1alpha1
  kind: Template
  metadata:
    name: git-pat-creds
  resources:
    manifests:
      - apiVersion: external-secrets.io/v1beta1
        kind: ExternalSecret
        metadata:
          name: git-pat-creds
        spec:
          dataFrom:
            - extract:
              conversionStrategy: Default
              key: git-pat-creds
          refreshInterval: 1m0s
          secretStoreRef:
            kind: SecretStore
            name: tenant-vault-shared-secret-store
          target:
            name: git-pat-creds

Now add a TemplateGroupInstance:

  apiVersion: tenantoperator.stakater.com/v1alpha1
  kind: TemplateGroupInstance
  metadata:
    name: git-pat-creds
  spec:
    template: git-pat-creds
    selector:
      matchExpressions:
        - key: stakater.com/kind
          operator: In
          values: [ build, pr ]
    sync: true

If you have correctly configured your infra repository, ArgoCD should be able to sync the changes and deploy the secret in build namespaces of the tenants

Repository Level Secrets#

[app-name]-ssh-creds
- Purpose: Used by these Tekton tasks:
  - git-clone
  - push-main-tag
  - create-git-tag
  - update-cd-repo
- Owner: Customer's delivery engineer
- Location: In build namespace of the tenant through apps-gitops repository
- Deployment Process: To deploy the git-pat-creds, follow the below-mentioned steps:
  1. Navigate to your apps-gitops-config repository
  2. Open up the tenant for which you want to deploy this secret.
  3. Now navigate to the folder which bears the name of the application for which you want to run the pipelines.
  4. Open the build folder.
  5. Add an external secret named [app-name]-ssh-creds:
```
  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: [app-name]-ssh-creds
  spec:
    secretStoreRef:
      name: tenant-vault-secret-store
      kind: SecretStore
    refreshInterval: "1m0s"
    target:
      name: [app-name]-ssh-creds
      creationPolicy: 'Owner'
      template:
        data:
          id_rsa: "{{ .id_rsa | b64dec | toString }}"
    data:
      - secretKey: id_rsa
        remoteRef:
          key: [app-name]-ssh-creds
          property: api_private_key
```
  6. Now open up the tenant path in Vault and add a secret named [app-name]-ssh-creds. Add a key api_private_key. The value should have a private ssh key that has access to your application repository as well as you apps-gitops-config repository.
  7. Assuming you have already set up the apps-gitops-config repository, you should be able to see the secret deployed to your tenant's build namespace

[app-name]-git-webhook-creds

Purpose: Used in the Repository CR. pipeline-as-code needs this to verify the webhook payload set
Owner: Developer owns this secret
Location: In build namespace of the tenant through apps-gitops repository

Deployment Process: Follow the below-mentioned steps for deploying the secret:

Navigate to your apps-gitops-config repository
Open up the tenant for which you want to deploy this secret.
Now navigate to the folder which bears the name of the application for which you want to run the pipelines.
Open the build folder.

Add an external secret named [app-name]-git-webhook-creds

  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: github-webhook-config
  spec:
    secretStoreRef:
      name: tenant-vault-secret-store
      kind: SecretStore
    refreshInterval: "1m0s"
    target:
      name: github-webhook-config
      creationPolicy: 'Owner'
      template:
        data:
          provider.token: "{{ .password | toString }}"
          webhook.secret: "{{ .secret | toString }}"
    data:
      - secretKey: password
        remoteRef:
          key: github-webhook-config
          property: provider.token
      - secretKey: secret
        remoteRef:
          key: github-webhook-config
          property: webhook.secret