Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

H-3710: Support DNS for HaRPC client and align environment variables #5856

Merged

Conversation

TimDiekmann
Copy link
Member

🌟 What is the purpose of this PR?

To allow HaRPC in AWS we need to allow DNS lookups in the client. This is needed to connect the different services as the IP of the graph could change with a redeployment and the DNS is stable. The situation is used to align the environment variables for the RPC and HTTP endpoints for the graph.

🔍 What does this change?

  • Use dns for the address for the HaRPC client
  • Enable HaRPC in dev-mode locally
  • Set an appropriated logging level
  • Align environment variables for the Graph
  • Prepare HaRPC on AWS (but keep it disabled)

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

❓ How to test this?

I did a lot of manual testing but I need the graph to be deployed in production to test this. As we use x86 in AWS I currently don't have a way to build those images locally and upload them without changing the AWS config.
There is a small chance that some environment variable was missed, but without a staging environment we cannot avoid this risk.

@TimDiekmann TimDiekmann self-assigned this Dec 11, 2024
@github-actions github-actions bot added area/apps > hash* Affects HASH (a `hash-*` app) area/infra Relates to version control, CI, CD or IaC (area) area/apps > hash-api Affects the HASH API (app) type/eng > backend Owned by the @backend team area/tests New or updated tests area/tests > integration New or updated integration tests area/apps area/apps > hash-graph area/infra > terraform labels Dec 11, 2024
Copy link

codecov bot commented Dec 11, 2024

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 22.97%. Comparing base (3790335) to head (314db98).
Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
apps/hash-api/src/generate-ontology-type-ids.ts 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5856      +/-   ##
==========================================
- Coverage   22.99%   22.97%   -0.03%     
==========================================
  Files         569      571       +2     
  Lines       19096    19298     +202     
  Branches     2709     2728      +19     
==========================================
+ Hits         4392     4433      +41     
- Misses      14652    14812     +160     
- Partials       52       53       +1     
Flag Coverage Δ
apps.hash-ai-worker-ts 1.32% <ø> (ø)
apps.hash-api 1.16% <0.00%> (ø)
blockprotocol.type-system 46.42% <ø> (ø)
local.harpc-client 62.15% <ø> (-5.99%) ⬇️
local.hash-backend-utils 8.80% <ø> (ø)
local.hash-graph-sdk 58.62% <ø> (ø)
local.hash-isomorphic-utils 0.99% <ø> (ø)
local.hash-subgraph 24.54% <ø> (ø)
rust.deer 6.66% <ø> (ø)
rust.error-stack 72.51% <ø> (ø)
rust.sarif 87.66% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

Benchmark results

@rust/hash-graph-benches – Integrations

representative_read_entity

Function Value Mean Flame graphs
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1 $$16.2 \mathrm{ms} \pm 170 \mathrm{μs}\left({\color{lightgreen}-29.417 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1 $$17.0 \mathrm{ms} \pm 251 \mathrm{μs}\left({\color{red}6.42 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1 $$18.0 \mathrm{ms} \pm 247 \mathrm{μs}\left({\color{red}12.6 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1 $$16.8 \mathrm{ms} \pm 191 \mathrm{μs}\left({\color{gray}-0.498 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1 $$16.4 \mathrm{ms} \pm 179 \mathrm{μs}\left({\color{gray}2.49 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2 $$17.4 \mathrm{ms} \pm 203 \mathrm{μs}\left({\color{red}6.16 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1 $$18.3 \mathrm{ms} \pm 225 \mathrm{μs}\left({\color{red}8.71 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1 $$16.3 \mathrm{ms} \pm 237 \mathrm{μs}\left({\color{lightgreen}-28.909 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1 $$17.5 \mathrm{ms} \pm 234 \mathrm{μs}\left({\color{gray}2.12 \mathrm{\%}}\right) $$ Flame Graph

representative_read_multiple_entities

Function Value Mean Flame graphs
entity_by_property depths: DT=255, PT=255, ET=255, E=255 $$67.1 \mathrm{ms} \pm 388 \mathrm{μs}\left({\color{gray}4.28 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=0, ET=0, E=0 $$38.1 \mathrm{ms} \pm 167 \mathrm{μs}\left({\color{gray}2.55 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=2, PT=2, ET=2, E=2 $$56.3 \mathrm{ms} \pm 252 \mathrm{μs}\left({\color{gray}1.21 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=0, ET=0, E=2 $$42.6 \mathrm{ms} \pm 313 \mathrm{μs}\left({\color{gray}0.981 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=0, ET=2, E=2 $$48.0 \mathrm{ms} \pm 187 \mathrm{μs}\left({\color{gray}-1.016 \mathrm{\%}}\right) $$ Flame Graph
entity_by_property depths: DT=0, PT=2, ET=2, E=2 $$53.0 \mathrm{ms} \pm 316 \mathrm{μs}\left({\color{gray}1.91 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=255, PT=255, ET=255, E=255 $$106 \mathrm{ms} \pm 600 \mathrm{μs}\left({\color{gray}0.474 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=0, ET=0, E=0 $$44.6 \mathrm{ms} \pm 203 \mathrm{μs}\left({\color{red}7.00 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=2, PT=2, ET=2, E=2 $$97.9 \mathrm{ms} \pm 689 \mathrm{μs}\left({\color{gray}0.279 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=0, ET=0, E=2 $$84.8 \mathrm{ms} \pm 409 \mathrm{μs}\left({\color{gray}2.35 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=0, ET=2, E=2 $$89.4 \mathrm{ms} \pm 425 \mathrm{μs}\left({\color{gray}-0.103 \mathrm{\%}}\right) $$ Flame Graph
link_by_source_by_property depths: DT=0, PT=2, ET=2, E=2 $$93.1 \mathrm{ms} \pm 355 \mathrm{μs}\left({\color{gray}-1.442 \mathrm{\%}}\right) $$ Flame Graph

representative_read_entity_type

Function Value Mean Flame graphs
get_entity_type_by_id Account ID: d4e16033-c281-4cde-aa35-9085bf2e7579 $$1.37 \mathrm{ms} \pm 3.55 \mathrm{μs}\left({\color{gray}-0.644 \mathrm{\%}}\right) $$ Flame Graph

scaling_read_entity_complete_one_depth

Function Value Mean Flame graphs
entity_by_id 50 entities $$273 \mathrm{ms} \pm 2.87 \mathrm{ms}\left({\color{gray}2.04 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 5 entities $$26.8 \mathrm{ms} \pm 304 \mathrm{μs}\left({\color{gray}0.398 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1 entities $$20.1 \mathrm{ms} \pm 89.6 \mathrm{μs}\left({\color{gray}-3.712 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$57.2 \mathrm{ms} \pm 295 \mathrm{μs}\left({\color{red}54.9 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 25 entities $$175 \mathrm{ms} \pm 1.08 \mathrm{ms}\left({\color{gray}-1.898 \mathrm{\%}}\right) $$ Flame Graph

scaling_read_entity_linkless

Function Value Mean Flame graphs
entity_by_id 1 entities $$1.91 \mathrm{ms} \pm 3.76 \mathrm{μs}\left({\color{gray}-0.042 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 100 entities $$2.12 \mathrm{ms} \pm 7.12 \mathrm{μs}\left({\color{red}5.00 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$1.93 \mathrm{ms} \pm 6.65 \mathrm{μs}\left({\color{gray}-0.846 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1000 entities $$2.88 \mathrm{ms} \pm 14.0 \mathrm{μs}\left({\color{gray}0.962 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10000 entities $$9.27 \mathrm{ms} \pm 79.0 \mathrm{μs}\left({\color{lightgreen}-31.068 \mathrm{\%}}\right) $$ Flame Graph

scaling_read_entity_complete_zero_depth

Function Value Mean Flame graphs
entity_by_id 50 entities $$4.02 \mathrm{ms} \pm 15.6 \mathrm{μs}\left({\color{lightgreen}-20.083 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 5 entities $$1.96 \mathrm{ms} \pm 8.26 \mathrm{μs}\left({\color{gray}-0.385 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 1 entities $$1.94 \mathrm{ms} \pm 11.6 \mathrm{μs}\left({\color{gray}0.715 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 10 entities $$2.15 \mathrm{ms} \pm 7.57 \mathrm{μs}\left({\color{gray}0.604 \mathrm{\%}}\right) $$ Flame Graph
entity_by_id 25 entities $$3.32 \mathrm{ms} \pm 15.9 \mathrm{μs}\left({\color{gray}2.47 \mathrm{\%}}\right) $$ Flame Graph

Copy link
Member

@indietyp indietyp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! 🎉 Minor remaining non-blocking questions

.env Show resolved Hide resolved
apps/hash-api/src/index.ts Show resolved Hide resolved
@TimDiekmann TimDiekmann added this pull request to the merge queue Dec 11, 2024
Merged via the queue into main with commit e453b1b Dec 11, 2024
165 of 166 checks passed
@TimDiekmann TimDiekmann deleted the t/h-3710-investigate-if-aws-allows-for-tcp-connections branch December 11, 2024 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/apps > hash* Affects HASH (a `hash-*` app) area/apps > hash-api Affects the HASH API (app) area/apps > hash-graph area/apps area/infra > terraform area/infra Relates to version control, CI, CD or IaC (area) area/tests > integration New or updated integration tests area/tests New or updated tests type/eng > backend Owned by the @backend team
Development

Successfully merging this pull request may close these issues.

2 participants