Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

creating a table configured with Apache Kafka #14715

Open
madRobot10 opened this issue Dec 25, 2024 · 1 comment
Open

creating a table configured with Apache Kafka #14715

madRobot10 opened this issue Dec 25, 2024 · 1 comment

Comments

@madRobot10
Copy link

I am trying to follow the doc. but I face this problem.

I created the topic in Kafka and inserted some data on it. also, the schema is created, but the table is not.

my schema:
{
"schemaName": "events",
"enableColumnBasedNullHandling": false,
"dimensionFieldSpecs": [
{
"name": "uuid",
"dataType": "STRING",
"notNull": false
}
],
"metricFieldSpecs": [
{
"name": "count",
"dataType": "INT",
"notNull": false
}
],
"dateTimeFieldSpecs": [
{
"name": "ts",
"dataType": "TIMESTAMP",
"notNull": false,
"format": "TIMESTAMP",
"granularity": "1:MILLISECONDS"
}
]
}

Table conf.:
{
"tableName": "events",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "ts",
"schemaName": "events",
"replicasPerPartition": "1"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.broker.list": "kafka:9092",
"stream.kafka.topic.name": "events",
"stream.kafka.consumer.type": "simple",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "10000"
}
},
"metadata": {
"customConfigs": {}
}
}
the error message:
{
"code": 500,
"error": "org.apache.pinot.spi.stream.TransientConsumerException: org.apache.pinot.shaded.org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 5000ms"
}

Kafka topic:
madhhar@madhharus:$ docker exec -it kafka kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events --from-beginning
{"ts": 1735112369985, "uuid": "c4456f4e-611b-48b6-b6da-550b4052c9d7", "count": 318}
{"ts": 1735112370987, "uuid": "90727af9-0914-40b3-8af6-65f1c678d676", "count": 911}
{"ts": 1735112371988, "uuid": "a280bbee-3e86-482f-839e-1872c0028995", "count": 768}
{"ts": 1735112372990, "uuid": "3ee49eb9-b1b3-40b0-8173-938f32887e3e", "count": 78}
{"ts": 1735112373991, "uuid": "065c78d7-c05d-4e9f-9d25-c5960b9d38cb", "count": 947}
{"ts": 1735112374992, "uuid": "3d811d72-050f-4b10-a5aa-e332f6acb411", "count": 450}
{"ts": 1735112375993, "uuid": "733eece5-2a57-48ef-a35c-0c3cf365f665", "count": 715}
{"ts": 1735112376995, "uuid": "6be53521-969c-401c-a27a-8c2ef19bc0a3", "count": 759}
{"ts": 1735112377995, "uuid": "b6075c33-50d8-4b5e-854a-45b14e0b9119", "count": 212}
{"ts": 1735112378996, "uuid": "86734094-8e15-4376-9c3e-b56618e2bb09", "count": 875}
{"ts": 1735112379998, "uuid": "fdfb2cce-bcce-4022-a3ee-827cb5f2170b", "count": 747}
{"ts": 1735112380999, "uuid": "0c45bcec-161a-412d-bf38-e1c92f716a80", "count": 547}
{"ts": 1735112382001, "uuid": "6fd0a307-d5d9-49fd-96af-67a5ce3f7d98", "count": 608}
{"ts": 1735112383002, "uuid": "04e4a8dc-1de0-4f1a-88a5-af7a343f59a4", "count": 46}
{"ts": 1735112384003, "uuid": "640944e6-abdf-49c5-8ef3-dd97e00dd676", "count": 224}
{"ts": 1735112385005, "uuid": "7d41bd43-145b-4606-891f-fe1d04c5ab78", "count": 238}
{"ts": 1735112386006, "uuid": "5a522e1c-cf89-47e4-9a51-07f9a800d3e0", "count": 244}
{"ts": 1735112387007, "uuid": "0048f77f-38f4-4765-9c83-7d447b421095", "count": 779}
{"ts": 1735112388008, "uuid": "31842f55-dd3a-44f8-beac-94fa7c7048a5", "count": 241}
{"ts": 1735112389010, "uuid": "bbc6b245-d964-45ac-a106-13e2702130ab", "count": 994}
{"ts": 1735112390011, "uuid": "1ad177f9-88ae-4559-8b70-bf37905d504e", "count": 122}
{"ts": 1735112391013, "uuid": "bf2b002e-130e-42f8-be62-459a520727e3", "count": 394}
{"ts": 1735112392014, "uuid": "d11d30f0-eeab-4efd-b94a-8a3258635178", "count": 600}
{"ts": 1735112393015, "uuid": "9dd78e7a-96c0-4050-add0-4cceb5bc6bb6", "count": 114}
{"ts": 1735112394016, "uuid": "6d3e629f-193a-400f-bd66-6d3fabeb1971", "count": 371}
{"ts": 1735112395018, "uuid": "14f78ecb-fa2a-4417-84dc-23e408cd6705", "count": 272}
{"ts": 1735112396019, "uuid": "fec5ca47-aa27-4a9b-851f-bcb275e1f3e7", "count": 323}
{"ts": 1735112397021, "uuid": "56979da0-a18e-4178-81b2-6fbc6b41b861", "count": 888}
{"ts": 1735112398022, "uuid": "1409767f-7745-4a70-a0e0-b23ef55888c8", "count": 658}
{"ts": 1735112399024, "uuid": "1905899b-406b-499b-8abd-58a24f553b1e", "count": 151}
{"ts": 1735112400025, "uuid": "45d6f861-5e94-4324-b6b9-d00e77cb60d9", "count": 433}
{"ts": 1735112401026, "uuid": "7aae3d5b-399f-4a90-9fad-9e4bab167d75", "count": 860}
{"ts": 1735112402028, "uuid": "19add349-d31c-44b3-ab8b-0cc1013bb037", "count": 300}
{"ts": 1735112403029, "uuid": "1a878f82-302a-4895-bced-ae4b7ba7f48d", "count": 992}
{"ts": 1735112404031, "uuid": "3dc66dad-e9e8-4ee6-95d9-d9a2e89c482d", "count": 352}
^CProcessed a total of 35 messages
madhhar@madhharus:
$

@Jackie-Jiang
Copy link
Contributor

Seems like a connection issue. Let me know if the problem is solved now.
You may get more help from the Slack community.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants