Market Data ETL Bootstrap

[abjjabjj]

• Token creation is reintroduced to agent builder
• Tokens are registered to market-data DB on launch
• Market Data TokenSymbol table is updated to include token address as a field
• Token financial data is scraped every minute via a background task on market data.

I think I may have shot myself in the foot here by choosing to create a separate database for market data. It theoretically makes it easier to isolate and manage the resources required for market data, but it also means we need to mirror/duplicate the token table on the API DB to the market data DB.

---

Important

Reintroduces token creation, updates market-data DB, and adds background task for token data scraping every minute.

• Behavior:
  • Reintroduces token creation in agent/builder/json.tsx and native.tsx.
  • Registers tokens to market-data DB on launch in bonding.ts.
  • Scrapes token financial data every minute via collect_timeseries in crud.py.
• Database:
  • Adds address field to TokenSymbol table in models.py and Alembic migration 6d6fe1c9f059_add_address_field_to_token_symbol_table.py.
• Configuration:
  • Updates NEXT_PUBLIC_MARKET_DATA_UPDATE_FREQUENCY to 60000 in .env.local.example and deploy-frontend-to-staging.yaml.
  • Adds SEI_RPC_URL and BONDING_CONTRACT_ADDRESS to market-data/.env.example.
• Dependencies:
  • Adds apscheduler and web3 to pyproject.toml for scheduling and blockchain interaction.

^{This description was created by for 7c45eec. You can customize this summary. It will automatically update as commits are pushed.}

[ellipsis-dev]

Important

Looks good to me! 👍

Reviewed everything up to ba5227f in 1 minute and 38 seconds. Click for details.

• Reviewed 2512 lines of code in 19 files
• Skipped 1 files when reviewing.
• Skipped posting 2 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. uv.lock:1

• Draft comment:
  The dependency lock file has been regenerated/updated with many new package entries (e.g. updated versions for web3, regex, pycryptodome, etc.). Please double‐check that these dependency version bumps and added wheels for various environments are intentional and tested across all target platforms.
• Reason this comment was not posted:
  Comment was not on a location in the diff, so it can't be submitted as a review comment.

2. apps/market-data/src/server.py:35

• Draft comment:
  Typo: The API title in FastAPI(title='AIDN Market Data API', ... ) might be intended to be 'Aiden' (matching repository name) unless this is intentional.
• Reason this comment was not posted:
  Comment was on unchanged code.

Workflow ID: wflow_MwgL3KAMjMYtoJxG

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

Important

Looks good to me! 👍

Reviewed aa08594 in 1 minute and 4 seconds. Click for details.

• Reviewed 13 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 1 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. apps/market-data/src/routers/udf.py:69

• Draft comment:
  Default exchange changed to 'DragonSwap'. Confirm this fallback is intentional and update related docs if needed.
• Reason this comment was not posted:
  Comment looked like it was already resolved.

Workflow ID: wflow_c8yYhYf64LB6F8Ud

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[ellipsis-dev]

Important

Looks good to me! 👍

Reviewed 7c45eec in 57 seconds. Click for details.

• Reviewed 24 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 2 draft comments. View those below.
• Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.

1. apps/frontend/lib/api/token.ts:18

• Draft comment:
  Good: 'address' field added. Ensure this field is validated and consistent with your DB schema.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50% None

2. apps/frontend/lib/contracts/bonding.ts:186

• Draft comment:
  Passing tokenAddress to createMarketDataToken is correct. Confirm that tokenAddress is always a valid contract address.
• Reason this comment was not posted:
  Confidence changes required: 0% <= threshold 50% None

Workflow ID: wflow_Md2FYczaYmmLvahK

^{You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.}

[kgang]

@crypto-cooker can you review this today

[crypto-cooker]

okay

[alpuga]

LGTM

[crypto-cooker]

Can you please attach the screenshot... @abjjabjj?

[abjjabjj]

https://www.loom.com/share/b12a8a059757472f9ecb273e2b737d46?sid=da3ca893-e413-4b91-9fbc-873e999e8c90

You can look at this video, the visual parts on the frontend that have changed is that the "Token" section has now returned to the agent builder

[kgang]

SO. COOL.

[kgang]

market-data needs to be deployed as its own service, so needs its own github workflow

[kgang]

im dumb we've already had it for two months

[kgang]

oh nice i see that andy deployed both the fe and new market-data service to staigen. I'm happy enough with how it is there, so i'm going to skip setting up my frontend (just to connect into staging's k8s at home anyway)

[kgang]

LGTM