Deployment Markers in New Relic for a Mono-Repo

We had an issue. We couldn’t track our deployments accurately across different domains. Sure, we could dive into GitHub and pipelines for details, but connecting the dots between performance issues and incidents was a real hassle. This caused delays and made it harder to make informed decisions.

The change tracking feature in New Relic is a powerful way to monitor the effects of specific changes on your customers and systems. This feature allows you to designate which changes you want to monitor and view the results in the New Relic UI.

Benefits of Deployment Markers

Using deployment markers in New Relic offers several advantages:

1. Improved Visibility: Deployment markers make it easier to track when and where changes were deployed. This helps in quickly identifying the impact of new deployments on your system's performance and stability.

2. Better Incident Response: By having clear markers, your team can quickly correlate issues with recent changes, reducing the mean time to resolve (MTTR) incidents.

3. Enhanced Accountability: Deployment markers provide a record of who deployed what and when. This helps in maintaining accountability and understanding the sequence of changes.

4. Seamless Monitoring: With deployment markers, you can monitor key performance indicators (KPIs) and other metrics before and after a deployment, allowing for data-driven decisions about rollbacks or further improvements.

Challenges and Considerations

While deployment markers offer significant benefits, there are some challenges to be aware of:

1. Configuration Complexity: Setting up deployment markers requires configuring your CI/CD pipeline and scripts, which can be time-consuming and may introduce complexity.

2. Data Overload: If not managed properly, deployment markers can generate a lot of data, making it challenging to sift through and find relevant information.

3. Consistency Issues: Ensuring that deployment markers are consistently applied across all services can be difficult, especially in large mono-repos.

Despite these challenges, the benefits of using deployment markers often outweigh the downsides, providing valuable insights into your deployment processes.

Setting Up New Relic CLI

docker-compose.yml

services:
  newrelic:
    image: newrelic/cli:latest
    env_file: ${ENVFILE:-.env}
    user: root
    volumes:
      - .:/opt/app

Searching for GUIDs

The newrelic-guid-search.sh script helps find the GUID for a specific service:

#!/bin/bash
RESULT="$(docker-compose run --rm newrelic entity search --name $1 --format Text --fields-filter guid --type AWSLAMBDAFUNCTION --case-sensitive true)" && \
GUID=$(echo "$RESULT" | tail -n +3 | head -n 1)
if [ -z "$GUID" ]; then
  echo "No New Relic GUID found for $1"
else
  echo "GUID found for $1: $GUID"
  export GUIDS=$(echo $GUIDS $GUID)
fi

Notifying Deployment

The Makefile contains the logic to notify New Relic about deployments:

define newrelicSearch
echo "Searching for GUIDs for $(SERVICES_LIST_FOR_DEPLOYMENT)"; \
echo "SERVICES_PREFIX $(SERVICES_PREFIX)"; \
GUIDS=""; \
for SERVICE in $(SERVICES_LIST_FOR_DEPLOYMENT) ; do \
  . ./scripts/newrelic-guid-search.sh $(SERVICES_PREFIX)-$(ENV)-$$SERVICE; \
done
endef

notifyDeploy: $(ENVFILE)
  @$(call newrelicSearch); \
  for GUID in

GUIDS ; do \
    docker compose run --rm newrelic entity deployment create --guid 

GUID --version $(VERSION) --commit

GO_REVISION_GIT --groupId 

GO_REVISION_GIT --user

GO_TRIGGER_USER --deepLink <https://github.com/<YOUR_REPO>/commit/>

GO_REVISION_GIT --description $(PROJECT_NAME):$(DOMAIN)-$(SERVICE); \
  done

Integrating with GoCD

The gocd.yml file integrates the deployment notification into the CI/CD pipeline:

# <https://github.com/tomzo/gocd-yaml-config-plugin>
format_version: 4
common:
  env_common: &env_common
  ...
  SERVICES_LIST_FOR_DEPLOYMENT: serviceOne serviceTwo
  secure_env: &secure_env
  ...
  NEW_RELIC_API_KEY: <NEWRELIC API KEY>
deploy_and_report_jobs: &deploy_and_report_jobs
deploy:
  tasks:
    - *fetch_node_modules_task
    - *fetch_artifact_task
    - exec:
      command: make
      arguments:
        - deploy
        - notifyDeploy
pipelines:
  service:
    group: pipeline-group
    ...
  stages:
    - deployToProd:
      ...
      environment_variables:
        <<: *env_common
      secure_variables:
        <<: *secure_env
      jobs:
        <<: *deploy_and_report_jobs

Visualizing Deployment Markers in New Relic

With our deployment notifications now integrated into New Relic, we can visualize the deployment markers directly within the New Relic console.

As illustrated in the New Relic console, deployment markers are prominently displayed along the timeline, indicating the exact time and date when deployments occurred. These markers provide immediate visual cues about changes in your application, making it easier to track and analyze the impact of each deployment.

Key Benefits:

1. Clear Chronology of Deployments:
   The deployment markers create a clear timeline of all changes deployed to your services. This allows teams to quickly pinpoint when a specific deployment took place, providing invaluable context during monitoring and troubleshooting activities.

2. Enhanced Error Correlation:
   By contrasting deployment markers with performance metrics and error rates, teams can readily correlate spikes or anomalies with recent deployments. This helps in diagnosing whether a new deployment might be the root cause of any sudden increases in errors or performance degradation.

3. Efficient Incident Management:
   Deployment markers assist in incident management by providing a historical record of deployments. When an incident arises, you can swiftly identify if a recent deployment might be contributing to the issue, thus reducing the time to resolution.

4. Improved Accountability and Auditing:
   Deployment markers also enhance accountability by providing a detailed audit trail of who deployed what and when. This information is crucial for post-deployment reviews and for ensuring that deployments are executed as planned.

Example in Console:

Here is an example of how deployment markers appear in the New Relic console:

Deployment Time and Date: Each marker shows the precise time and date of the deployment, helping in maintaining a chronological order of events.

Associated Changes: Hovering over a marker provides detailed information about the deployment, including the version deployed, commit details, and the responsible user.

This visual representation not only aids in monitoring the health of your applications but also ensures that any changes that lead to potential issues are easily identifiable and can be addressed promptly. An increase in errors after a deployment may indicate a bug in the deployed code.

By integrating deployment markers into your monitoring setup, you gain a comprehensive view of your application’s lifecycle, enhancing both the reliability and transparency of your deployment process.

Conclusion

As discussed in my previous posts on using GitHub Actions to trigger actions across repositories and enabling code reuse with a mono-repo approach, setting up a robust CI/CD pipeline and having a mono-repo structure greatly improves efficiency and maintainability. By integrating deployment markers, we further enhance our ability to track changes and their impacts, ensuring that our projects are not only scalable but also maintainable and transparent.

However, it's important to note that this solution is currently not applicable to app deployments. Each of the app stores—such as the Apple App Store and Google Play Store—requires a manual release process. This means we cannot automatically notify New Relic when a new app version is released. Instead, we rely on version tags to monitor the usage and adoption of each version.

This method is effective because an app release does not immediately translate to all customers using the new version. Users must update the app on their devices, which introduces a delay between the release and widespread usage. By tracking version tags, we can still monitor the performance and usage of different app versions, ensuring that any issues related to specific versions can be promptly identified and addressed.

Ultimately, integrating deployment markers and version tags allows us to maintain high visibility over our deployments and user interactions, ensuring a smoother and more efficient release process across all platforms.

The views expressed on this blog post are mine alone and do not necessarily reflect the views of my employer, Optus Administration Pty Ltd.