[source-zendesk-support] macro stream pagination is not supported by Airbyte UI.

[Himanshu-Dharma]

Topic

Zendesk Support macro ingestion

Relevant information

Hello team,

At present,

• Zendesk Support is available as a connector support by Airbyte. Within Zendesk Support, there is a macros stream, wherein the data is limited to 100 per page.
• Airbyte does not support the pagination for this stream.

I wanted to request, if this can be implemented from Airbyte end or if it planned to be rolled out somewhere in the future.

Following is the documentation,

• https://developer.zendesk.com/api-reference/ticketing/business-rules/macros/
• https://developer.zendesk.com/api-reference/introduction/pagination/

Reason for the request,

• It is just this single stream because if which, we might need to go for wither a custom connector approach or some other ingestion tool. It would be really helpful if there could be some sort of a resolution on this. As zendesk and it's streams are widely used, and it also has quite a number of stream.

Thank you!!

[marcosmarxm]

Hello @Himanshu-Dharma the current implementation of the connector already support pagination:

airbyte/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/manifest.yaml

Lines 289 to 306 in 3aa1297

	macros_stream:
	$ref: "#/definitions/base_incremental_stream"
	retriever:
	$ref: "#/definitions/semi_incremental_stream/retriever"
	requester:
	$ref: "#/definitions/retriever/requester"
	request_parameters:
	sort_by: "created_at"
	sort_order: "asc"
	paginator:
	$ref: "#/definitions/links_next_paginator"
	incremental_sync:
	$ref: "#/definitions/semi_incremental_stream/incremental_sync"
	$parameters:
	name: "macros"
	path: "macros"
	cursor_field: "updated_at"
	primary_key: "id"

Paginator:

airbyte/airbyte-integrations/connectors/source-zendesk-support/source_zendesk_support/manifest.yaml

Lines 112 to 125 in 3aa1297

	# Paginators
	links_next_paginator:
	type: DefaultPaginator
	pagination_strategy:
	type: CursorPagination
	cursor_value: "{{ response['links']['next'] }}"
	stop_condition: "{{ not response['meta']['has_more'] }}"
	page_size: 100
	page_token_option:
	type: RequestPath
	page_size_option:
	type: RequestOption
	field_name: "page[size]"
	inject_into: request_parameter

The paginator follows instructions from https://developer.zendesk.com/api-reference/introduction/pagination/
If pagination is not work there are some investigation to be done why this is happening.

[marcosmarxm]

@Himanshu-Dharma can you please share what version of the connector are you usin?

[Himanshu-Dharma]

Hello @marcosmarxm, Thank you so much for taking this and updating me. Really appreciate it.
Connector version we're using: Zendesk Support v2.2.8