[CT-2184] [Bug] null values should be of type null and not of type string in manifest.json

[geoHeil]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Null values are rendered as proprietary "None" string fields in manifest.json.

https://getdbt.slack.com/archives/C01PWH4TXLY/p1677050123391359

Expected Behavior

Null values are rendered as valid JSON nulls in manifest.json

Steps To Reproduce

dbt run

for version 1.3.2 and also 1.4.1

Relevant log output

No response

Environment

- OS: osx, linux
- Python:3.10
- dbt: 1.3.2, 1.4.1

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

No response

[geoHeil]

oracle/dbt-oracle#74 and open-metadata/OpenMetadata#10282 for further reference and implementation of the workaround

[jtcohen6]

Context from Slack:

I am using DBT 1.3.2 with DBT-oracle 1.3.2 and observe in the manifest.json that the database field for the nodes is showing as “None” in a string instead of null is this an issue with DBT it self or dbt-oracle? How can I get it to return valid null values?

And from the linked issue in oracle/dbt-oracle#74 (comment):

Python None type translates to null in JSON

The issue you reported is only the case with database field because of how database is declared. We could not declare the default value to None as you can see below.

   # Specifying database is optional
   database: Optional[str]

   # OracleConnectionMethod.TNS
   tns_name: Optional[str] = None

The reason is some validation enforced by dbt. if you change database's default value to None, dbt command fails.

File "/usr/local/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/dataclasses.py", line > 504, in _init_fn
   raise TypeError(f'non-default argument {f.name!r} '
TypeError: non-default argument 'schema' follows default argument

If default value could be declared None, manifest.json will have a valid null

Oracle is one of a handful of databases that only support a two-level namespace, rather than a three-level, so dbt needs to ignore the existence of database (canonical name for the third level). Apache Spark was the first data platform with a dbt integration where this was also the case. I believe we have some logic in dbt-spark that enables database to be set as actual Python None, rather than the string "None", through a couple of post-init hooks on the relevant dataclasses. I don't know if these are all of the relevant spots, but most of them:

• SparkCredentials. __pre_deserialize__
• get_relation
• SparkRelation

So I'm inclined to believe that, if dbt-oracle were to take a slightly different approach, it could change the internal representation of the unused database field to true Python None, rather than the string "None".

I'm going to close this issue in the meantime, but we can reopen if the dbt-oracle maintainers believe that there's a specific change in dbt-core that would make this possible/easier.