Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev minor #1423

Merged
merged 7 commits into from
Oct 17, 2024
Merged

Dev minor #1423

merged 7 commits into from
Oct 17, 2024

Conversation

emrgnt-cmplxty
Copy link
Contributor

@emrgnt-cmplxty emrgnt-cmplxty commented Oct 17, 2024

Important

Enhancements to ingestion, document chunk vector inclusion, prompt handling, and knowledge graph testing, with new tests added for KG logic and integration.

  • Ingestion:
    • Added support for multimodal foundation models for PDFs in ingestion.mdx and js-sdk/ingestion.mdx.
    • Updated ingestion configuration options in parsing_and_chunking.mdx.
  • Document Chunks:
    • Added include_vectors option in management.py and management_router.py to include vectors in document chunks.
  • Prompt Handling:
    • Made get_prompt and _get_message_payload asynchronous in prompt.py and r2r_prompts.py.
  • Knowledge Graph:
    • Added tests for KG logic in test_kg_logic.py.
    • Updated KG provider to handle embeddings as strings in postgres.py.
  • Testing:
    • Added integration tests for KG creation and enrichment in runner_cli.py and runner_sdk.py.
    • Updated main.py to automate ingestion and graph creation processes.

This description was created by Ellipsis for 11ae42e. It will automatically update as commits are pushed.

emrgnt-cmplxty and others added 6 commits October 16, 2024 15:04
* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

* some various documentation tweaks

* some various documentation tweaks

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* modify timeouts

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* feat: Make prompt provider methods asynchronous

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Copy link

vercel bot commented Oct 17, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
yc_demo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 17, 2024 1:40am
yc-demo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 17, 2024 1:40am
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
recommendation_platform ⬜️ Ignored (Inspect) Oct 17, 2024 1:40am

@emrgnt-cmplxty emrgnt-cmplxty marked this pull request as ready for review October 17, 2024 01:39
@emrgnt-cmplxty emrgnt-cmplxty merged commit e3434b6 into main Oct 17, 2024
20 of 73 checks passed
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Reviewed everything up to 11ae42e in 1 minute and 15 seconds

More details
  • Looked at 1610 lines of code in 39 files
  • Skipped 0 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. templates/ycombinator_graphrag/python-backend/main.py:104
  • Draft comment:
    The calls to ingest_data() and create_graph() are commented out. If this is intentional for testing, ensure to uncomment them for production use.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The ingest function is currently calling ingest_data, create_graph, and enrich_graph in sequence. However, in the diff, the calls to ingest_data and create_graph are commented out. This might be intentional for testing purposes, but it should be noted.

Workflow ID: wflow_FcwGFJuqrK7BXQPS


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

@shreyaspimpalgaonkar shreyaspimpalgaonkar deleted the dev-minor branch October 17, 2024 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants