Skip to content
This repository was archived by the owner on Dec 4, 2025. It is now read-only.

Add prometheus monitoring to frontend#35

Draft
MichaelDeng03 wants to merge 27 commits intomainfrom
michael/prometheus-frontend
Draft

Add prometheus monitoring to frontend#35
MichaelDeng03 wants to merge 27 commits intomainfrom
michael/prometheus-frontend

Conversation

@MichaelDeng03
Copy link
Copy Markdown
Contributor

@MichaelDeng03 MichaelDeng03 commented Feb 28, 2025

What's been implemented:
Prometheus exporter using open telemetry so that prometheus backend in apps/prometheus can scrape the frontend's /metrics endpoint.

Info dump:
Predominantly yoinked from this dudes solution. Relevant parent discussion regarding sticking a prometheus exporter into a next.js server
Next.js version 15 makes telemetry stable, so you don't need to configure nextConfig w/ experimental telemetry hook.
I used a similar solution in eliza
Next.js documentation on adding instrumentation
Why prom-client was not used

Quickstart:

make run-prometheus
make run-frontend # couldn't get get outside of docker to work, will look at this l8r

prometheus will be available at http://localhost:9090
make sure that frontend was discovered in the status -> service discovery drop down

you can run some queries on prometheus about your frontend:
{job="frontend"} will list all metrics (go to table view)
example metric w/ nothing fancy:
nodejs_eventloop_time will graph your eventloop time (go to the graph)

TODO:

  • Update prometheus backend app to see frontend
  • Update aws (holy friggin frick)
    • Figure out how to expose and restrict port 9464

Squashed commit of the following:

commit 7caf024
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 15:51:42 2025 -0600

    Update eliza to use latest aiden-stable w/ braintrust integration

commit e3e0d9c
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 15:42:30 2025 -0600

    Remove otel instrumentator. Lock prometheus instrumentator version

commit caa6d9a
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 15:39:42 2025 -0600

    Add gha for deploying prometheus to prod

commit 7c9ae67
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 15:06:31 2025 -0600

    Fix prometheus config for prod

commit def172c
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 12:25:10 2025 -0600

    Add GitHub Actions workflow for deploying Prometheus to staging

commit 062e0af
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 12:24:31 2025 -0600

    Rename stage environment from stage -> staging and update Prometheus listen address to bind to all ports

commit 985e55e
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 12:24:02 2025 -0600

    Name prometheus image

commit 4c46596
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 10:17:58 2025 -0600

    Add AWS ECR push command for Prometheus in Makefile

commit df28165
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 10:17:52 2025 -0600

    Add TODO comments for frontend and agent-runtime scrape configs in Prometheus YAML files

commit 96382ef
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 10:09:24 2025 -0600

    Add Prometheus configuration files for development and staging and prod envs.

commit 3873332
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 10:08:55 2025 -0600

    Move fastapi prometheus instrumentation outside of lifespan

commit 6e4842b
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 10:08:10 2025 -0600

    Update API port to be consistent, and not in conflict with frontend/runtime (to 8003)

commit 758b9a3
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 10:07:36 2025 -0600

    Add support for using different prometheus config depending on env.

commit c16a281
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 09:06:18 2025 -0600

    Add prometheus' fastapi instrumentor

commit d65e9a4
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 08:59:14 2025 -0600

    Add prometheus as an app

commit a0539b4
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Fri Feb 28 08:27:47 2025 -0600

    Add .env.example to apps/runtime w/ info about observability for eliza agents

commit 2e05cea
Author: Michael Deng <michaeldeng2003@gmail.com>
Date:   Wed Feb 26 17:10:16 2025 -0600

    Add otel and auto instrument api server
@MichaelDeng03 MichaelDeng03 marked this pull request as draft March 1, 2025 00:07
@MichaelDeng03 MichaelDeng03 requested review from abjjabjj and kgang and removed request for kgang March 1, 2025 00:07
Copy link
Copy Markdown
Contributor Author

monsieur @abjjabjj if you could take a look at the instrumentation in this pr when you get back, that would be great
it's missing a few things, but available for playing around with locally
Here's a quickstart, I'll update the pr comment with this as well

make run-prometheus
make run-frontend # couldn't get get outside of docker to work, will look at this l8r

prometheus will be available at http://localhost:9090
make sure that frontend was discovered in the status -> service discovery drop down

you can run some queriers about your frontend,
{job="frontend"} will list all metrics (go to table view)
nodejs_eventloop_time will graph your eventloop time (go to the graph)

@MichaelDeng03 MichaelDeng03 changed the title Michael/prometheus frontend Add prometheus monitoring to frontend Mar 3, 2025
Copy link
Copy Markdown
Contributor Author

ill put a pr for this into main after 34 is merged
there's a few more snags that I have to work through tho

Base automatically changed from michael/observability to main March 3, 2025 23:32
@MichaelDeng03 MichaelDeng03 linked an issue Mar 12, 2025 that may be closed by this pull request
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prometheus observability onto frontend

1 participant