Expose Exclude Status and Doc-type via the API

[CarsonDavis]

Description

When the JSON connector indexes via the webapp api, it needs to only index the urls marked as indexable. I'm thinking we make this a parameter passed to the api, so that you tell it you want only indexable urls, and it only returns indexable. Additionally, there will be a metadata field that marks whether the url is excluded or not, in case someone wants to do a post query filter.

Additionally, we need to check that doc type is supplied for every url, so the json connector can store it.

Implementation Considerations

• type your first consideration here

Deliverable

Made changes to make sure that the API serves excluded/not excluded field via the endpoint for every url when called.