Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add migration scripts to export concepts and import them to another Leaf instance #12

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

rpkyle
Copy link
Contributor

@rpkyle rpkyle commented Oct 25, 2024

This pull request aims to contribute the following assets to streamline concept migration from one Leaf instance to another:

  • sp_generate_inserts.sql -- This stored procedure was obtained from here with corrections applied as suggested here. The procedure generates an SQL insert statement from an existing SQL table so that the data contained within it can be loaded into another database.
  • concept-cloner.sh -- This shell script calls generate_inserts.sh, which exports the app.Specialization, app.Concept, app.ConceptSqlSet, and app.SpecializationGroup tables using sp_generate_inserts, then copies the resulting INSERT statements into the Docker container specified by the argument provided to concept-cloner.sh. It also fixes up the output so that any single quotes are replaced with backticks, and trailing whitespace is removed when present inside of quoted strings.
  • concept-replace.sh -- This script copies the INSERT statements generated by concept-cloner.sh into the Docker container, deletes old concepts and relationship information, then pushes the new concepts using the insert_concepts.sh shell script.
  • insert_concepts.sh -- this is a helper script that repeatedly calls sqlcmd to remove and replace the required concept and specialization data.
  • generate_inserts.sh -- this is a helper script that repeatedly calls sqlcmd to run sp_generate_inserts to export the required tables, and uses sed to fix up the output to remove trailing whitespace.

The usage is simple. The scripts use the Docker service name and extract the container name from docker compose ps -a, since it may be easier to recall the service name versus the container name:

./concept-cloner.sh leaf-env-prod-gateway-mssql-1
./concept-replace.sh jhhcc-mssql

In addition, the R script used to produce SQL tables from data.frame objects is also contributed here as generate_inserts.R, based on code posted here.

Note that the original version of the R functions produces output that has <SQL> in the header line, which must be removed. Additionally, the resulting statements are not terminated with semicolons, and single quotes are used where backticks may be necessary. This can be fixed for now using sed like so:

sed -i "s,\","\`",g" output.sql

@rpkyle rpkyle requested a review from ivan-c October 25, 2024 04:55
@rpkyle rpkyle self-assigned this Oct 25, 2024
@rpkyle rpkyle requested review from ivan-c and removed request for ivan-c December 12, 2024 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant