Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expected datetime: found "2024-12-30T16:01:05.978665" #1190

Open
muhammad-levi opened this issue Sep 19, 2024 · 2 comments
Open

Expected datetime: found "2024-12-30T16:01:05.978665" #1190

muhammad-levi opened this issue Sep 19, 2024 · 2 comments

Comments

@muhammad-levi
Copy link
Collaborator

muhammad-levi commented Sep 19, 2024 •
edited
Loading

I do not know how to find the root cause(s), but it seems that some fields are required to be of type datetime when fhir-data-pipes tried to POST the transaction bundle into a Google Healthcare API FHIR Data Store.

The error log was:

07:24:27.092 [CHAIN DataSource (at Create.Values2/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search).output) (2/2)#0] WARN  o.a.flink.runtime.taskmanager.Task org.apache.flink.runtime.taskmanager.Task.transitionState:1097 - CHAIN DataSource (at Create.Values2/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources2/ParDo(Search)/ParMultiDo(Search).output) (2/2)#0 (ee03a40d89eb96ab43031516685244d0) switched from RUNNING to FAILED with failure cause: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-12-30T16:01:05.978665"

cause: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-12-30T16:01:05.978665"
...
Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-12-30T16:01:05.978665"

Full stack trace:

2024-09-20 09:34:42 
2024-09-20 09:34:42   .   ____          _            __ _ _
2024-09-20 09:34:42  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
2024-09-20 09:34:42 ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
2024-09-20 09:34:42  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
2024-09-20 09:34:42   '  |____| .__|_| |_|_| |_\__, | / / / /
2024-09-20 09:34:42  =========|_|==============|___/=/_/_/_/
2024-09-20 09:34:42  :: Spring Boot ::                (v3.1.5)
2024-09-20 09:34:42 
2024-09-20 09:34:42 02:34:42.667 [main] INFO  c.g.f.a.ControlPanelApplication org.springframework.boot.StartupInfoLogger.logStarting:51 - Starting ControlPanelApplication using Java 17.0.6 with PID 8 (/app/controller-bundled.jar started by root in /app)
2024-09-20 09:34:42 02:34:42.670 [main] INFO  c.g.f.a.ControlPanelApplication org.springframework.boot.SpringApplication.logStartupProfileInfo:638 - No active profile set, falling back to 1 default profile: "default"
2024-09-20 09:34:43 Sep 20, 2024 2:34:43 AM org.apache.coyote.AbstractProtocol init
2024-09-20 09:34:43 INFO: Initializing ProtocolHandler ["http-nio-8080"]
2024-09-20 09:34:43 Sep 20, 2024 2:34:43 AM org.apache.catalina.core.StandardService startInternal
2024-09-20 09:34:43 INFO: Starting service [Tomcat]
2024-09-20 09:34:43 Sep 20, 2024 2:34:43 AM org.apache.catalina.core.StandardEngine startInternal
2024-09-20 09:34:43 INFO: Starting Servlet engine: [Apache Tomcat/10.1.15]
2024-09-20 09:34:44 Sep 20, 2024 2:34:44 AM org.apache.catalina.core.ApplicationContext log
2024-09-20 09:34:44 INFO: Initializing Spring embedded WebApplicationContext
2024-09-20 09:34:44 02:34:44.132 [main] WARN  o.s.boot.actuate.endpoint.EndpointId o.springframework.boot.actuate.endpoint.EndpointId.logWarning:155 - Endpoint ID 'pipeline-metrics' contains invalid characters, please migrate to a valid format.
2024-09-20 09:34:44 02:34:44.186 [main] WARN  o.s.c.LocalVariableTableParameterNameDiscoverer o.s.core.LocalVariableTableParameterNameDiscoverer.inspectClass:123 - Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: com.google.fhir.analytics.DataProperties$ConfigFields
2024-09-20 09:34:44 02:34:44.208 [main] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.validateProperties:138 - Using FHIR-search mode since dbConfig is not set.
2024-09-20 09:34:44 02:34:44.222 [main] INFO  c.g.f.analytics.FlinkConfiguration com.google.fhir.analytics.FlinkConfiguration.initialiseFlinkConfiguration:122 - Flink configuration will be initialised from the directory at /app/config
2024-09-20 09:34:44 02:34:44.290 [main] INFO  ca.uhn.fhir.util.VersionUtil ca.uhn.fhir.util.VersionUtil.initialize:84 - HAPI FHIR version 6.10.4 - Rev 5d6bb47853
2024-09-20 09:34:44 02:34:44.301 [main] INFO  ca.uhn.fhir.context.FhirContext ca.uhn.fhir.context.FhirContext.<init>:226 - Creating new FHIR context for FHIR version [R4]
2024-09-20 09:34:44 02:34:44.356 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-resources.xml
2024-09-20 09:34:44 02:34:44.389 [main] INFO  ca.uhn.fhir.util.XmlUtil ca.uhn.fhir.util.jar.DependencyLogImpl.logStaxImplementation:76 - FHIR XML procesing will use StAX implementation 'Woodstox' version '5.4.0'
2024-09-20 09:34:45 02:34:45.437 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-types.xml
2024-09-20 09:34:45 02:34:45.472 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-others.xml
2024-09-20 09:34:45 02:34:45.566 [main] INFO  c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:368 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/extension/extension-definitions.xml
2024-09-20 09:34:46 02:34:46.526 [main] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:34:47 02:34:47.519 [main] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:34:47 02:34:47.522 [main] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.testFhirConnection:324 - Validating FHIR connection
2024-09-20 09:34:48 02:34:48.562 [main] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.testFhirConnection:335 - Validating FHIR connection successful
2024-09-20 09:34:48 02:34:48.622 [main] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.getAllChildDirectories:378 - Child directories : [/dwh/controller_DWH_TIMESTAMP_2024_09_19T06_34_31_302626884Z/, /dwh/controller_DWH_TIMESTAMP_2024_09_19T03_03_51_972577779Z/]
2024-09-20 09:34:48 02:34:48.655 [main] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.initDwhStatus:232 - Initializing with most recent DWH controller_DWH_TIMESTAMP_2024_09_19T06_34_31_302626884Z
2024-09-20 09:34:48 02:34:48.941 [main] WARN  o.s.c.LocalVariableTableParameterNameDiscoverer o.s.core.LocalVariableTableParameterNameDiscoverer.inspectClass:123 - Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: com.google.fhir.analytics.metrics.PipelineMetricsEndpoint
2024-09-20 09:34:49 Sep 20, 2024 2:34:49 AM org.apache.coyote.AbstractProtocol start
2024-09-20 09:34:49 INFO: Starting ProtocolHandler ["http-nio-8080"]
2024-09-20 09:34:49 02:34:49.113 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:119 - Last purge run was at null next run is at 2024-09-20T02:34:49.113309176
2024-09-20 09:34:49 02:34:49.113 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:121 - Purge run triggered at 2024-09-20T02:34:49.113862832
2024-09-20 09:34:49 02:34:49.114 [main] INFO  c.g.f.a.ControlPanelApplication org.springframework.boot.StartupInfoLogger.logStarted:57 - Started ControlPanelApplication in 7.082 seconds (process running for 7.985)
2024-09-20 09:34:49 02:34:49.144 [main] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.onApplicationEvent:501 - createHiveResourceTables is false; skipping Hive table creation.
2024-09-20 09:34:49 02:34:49.166 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.getAllChildDirectories:378 - Child directories : [/dwh/controller_DWH_TIMESTAMP_2024_09_19T06_34_31_302626884Z/, /dwh/controller_DWH_TIMESTAMP_2024_09_19T03_03_51_972577779Z/]
2024-09-20 09:34:49 02:34:49.183 [scheduling-1] INFO  c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:123 - Purge run completed at 2024-09-20T02:34:49.183598639
2024-09-20 09:34:49 02:34:49.184 [scheduling-1] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:373 - Last run was at 2024-09-20T02:34:48.655980377 next run is at 2024-09-20T02:45
2024-09-20 09:35:01 Sep 20, 2024 2:35:01 AM org.apache.catalina.core.ApplicationContext log
2024-09-20 09:35:01 INFO: Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-09-20 09:35:01 02:35:01.637 [http-nio-8080-exec-1] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:01 02:35:01.661 [http-nio-8080-exec-1] INFO  c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
2024-09-20 09:35:03 02:35:03.005 [http-nio-8080-exec-3] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:03 02:35:03.007 [http-nio-8080-exec-3] INFO  c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
2024-09-20 09:35:06 02:35:06.629 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.ApiController com.google.fhir.analytics.ApiController.runBatch:66 - Received request to start the pipeline ...
2024-09-20 09:35:06 02:35:06.630 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:06 02:35:06.640 [http-nio-8080-exec-5] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:06 02:35:06.841 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:221 - Fetching first batch of Flag
2024-09-20 09:35:06 02:35:06.996 [http-nio-8080-exec-5] INFO  com.google.fhir.analytics.FetchUtil c.u.f.rest.client.interceptor.LoggingInterceptor.interceptRequest:82 - Client request: GET https://fhir-server.dev.sid-indonesia.org/fhir/Flag?_count=100&_total=accurate&_summary=data HTTP/1.1
2024-09-20 09:35:07 02:35:07.417 [http-nio-8080-exec-5] INFO  com.google.fhir.analytics.FetchUtil c.u.f.rest.client.interceptor.LoggingInterceptor.interceptResponse:127 - Client response: HTTP 200 OK in 420ms
2024-09-20 09:35:07 02:35:07.495 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:239 - Number of resources for Flag search is 177
2024-09-20 09:35:07 02:35:07.497 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:255 - Total number of segments for resource Flag is 2
2024-09-20 09:35:08 02:35:08.648 [http-nio-8080-exec-5] INFO  c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.runBatchPipeline:418 - Running full pipeline for DWH /dwh/controller_DWH_TIMESTAMP_2024_09_20T02_35_06_630731647Z
2024-09-20 09:35:08 02:35:08.648 [Thread-3] INFO  c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:653 - Pipelines execution started with a new thread; number of threads is 20
2024-09-20 09:35:08 02:35:08.681 [pool-5-thread-1] WARN  o.a.beam.runners.flink.FlinkRunner org.apache.beam.runners.flink.FlinkRunner.run:93 - For maximum performance you should set the 'fasterCopy' option. See more at https://issues.apache.org/jira/browse/BEAM-11146
2024-09-20 09:35:10 02:35:10.577 [pool-5-thread-1] WARN  o.a.f.r.webmonitor.WebMonitorUtils o.a.f.r.webmonitor.WebMonitorUtils$LogFileLocation.find:82 - Log file environment variable 'log.file' is not set.
2024-09-20 09:35:10 02:35:10.577 [pool-5-thread-1] WARN  o.a.f.r.webmonitor.WebMonitorUtils o.a.f.r.webmonitor.WebMonitorUtils$LogFileLocation.find:88 - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'web.log.path'.
2024-09-20 09:35:10 02:35:10.664 [http-nio-8080-exec-8] INFO  c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:179 - Converting options for fhirServerUrl https://fhir-server.dev.sid-indonesia.org/fhir and dbConfig null
2024-09-20 09:35:10 02:35:10.668 [http-nio-8080-exec-8] INFO  c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
2024-09-20 09:35:11 02:35:11.045 [pool-5-thread-1] INFO  c.g.f.a.metrics.FlinkJobListener c.google.fhir.analytics.metrics.FlinkJobListener.onJobSubmitted:37 - Submitting the job with ID com.google.fhir.analytics.metrics.FlinkJobListener@1344cf21 
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (2/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (3/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.406 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (5/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.407 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (1/5)#0] INFO  com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.<init>:82 - Fetching access tokens from https://sso.dev.sid-indonesia.org/auth/realms/team-based-care/protocol/openid-connect/token
2024-09-20 09:35:11 02:35:11.449 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] INFO  c.g.fhir.analytics.FetchResources com.google.fhir.analytics.FetchResources$SearchFn.processElement:138 - Fetching 100 resources for state Flag_main; URL= https://fhir-server.dev.sid-indonesia.org/fhir?_getpages=2e246ea1-ae96-46b6-a15a-44f485e6dea9&_getpagesoffset=100
2024-09-20 09:35:11 02:35:11.451 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (5/5)#0] INFO  c.g.fhir.analytics.FetchResources com.google.fhir.analytics.FetchResources$SearchFn.processElement:138 - Fetching 100 resources for state Flag_main; URL= https://fhir-server.dev.sid-indonesia.org/fhir?_getpages=2e246ea1-ae96-46b6-a15a-44f485e6dea9&_getpagesoffset=0
2024-09-20 09:35:12 02:35:12.438 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (5/5)#0] INFO  c.google.fhir.analytics.GcpStoreUtil com.google.fhir.analytics.GcpStoreUtil.uploadBundle:79 - Full URL is: https://healthcare.googleapis.com/v1/projects/nextgen-398301/locations/asia-southeast2/datasets/team-based-care.dev/fhirStores/team-based-care.dev.data-store/fhir
2024-09-20 09:35:12 02:35:12.438 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] INFO  c.google.fhir.analytics.GcpStoreUtil com.google.fhir.analytics.GcpStoreUtil.uploadBundle:79 - Full URL is: https://healthcare.googleapis.com/v1/projects/nextgen-398301/locations/asia-southeast2/datasets/team-based-care.dev/fhirStores/team-based-care.dev.data-store/fhir
2024-09-20 09:35:15 02:35:15.300 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] ERROR o.a.f.runtime.operators.BatchTask org.apache.flink.runtime.operators.BatchTask.logAndThrowException:1325 - Error in task code:  CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)
2024-09-20 09:35:15 org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.305 [CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0] WARN  o.a.flink.runtime.taskmanager.Task org.apache.flink.runtime.taskmanager.Task.transitionState:1097 - CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search)) -> FlatMap (FlatMap at FetchResources/ParDo(Search)/ParMultiDo(Search).output) (4/5)#0 (c4ac18626093cabc98c1ac4eab54d3ac) switched from RUNNING to FAILED with failure cause: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 
2024-09-20 09:35:15 02:35:15.601 [pool-5-thread-1] ERROR c.g.f.a.metrics.FlinkJobListener c.google.fhir.analytics.metrics.FlinkJobListener.onJobExecuted:49 - Error while executing the job
2024-09-20 09:35:15 org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.601 [pool-5-thread-1] ERROR o.a.beam.runners.flink.FlinkRunner org.apache.beam.runners.flink.FlinkRunner.run:106 - Pipeline execution failed
2024-09-20 09:35:15 org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.602 [Thread-3] ERROR com.google.fhir.analytics.EtlUtils com.google.fhir.analytics.EtlUtils.runMultiplePipelines:141 - Error while executing the pipelines
2024-09-20 09:35:15 java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:129)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelinesWithTimestamp(EtlUtils.java:78)
2024-09-20 09:35:15     at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:667)
2024-09-20 09:35:15 Caused by: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:163)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:153)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.602 [Thread-3] ERROR c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:701 - exception while running pipeline: 
2024-09-20 09:35:15 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:142)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelinesWithTimestamp(EtlUtils.java:78)
2024-09-20 09:35:15     at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:667)
2024-09-20 09:35:15 Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:129)
2024-09-20 09:35:15     ... 2 common frames omitted
2024-09-20 09:35:15 Caused by: java.lang.RuntimeException: Pipeline execution failed
2024-09-20 09:35:15     at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
2024-09-20 09:35:15     at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:163)
2024-09-20 09:35:15     at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:153)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2024-09-20 09:35:15     at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2024-09-20 09:35:15     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:300)
2024-09-20 09:35:15     at akka.dispatch.OnComplete.internal(Future.scala:297)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
2024-09-20 09:35:15     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2024-09-20 09:35:15     at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
2024-09-20 09:35:15     at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
2024-09-20 09:35:15     at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2024-09-20 09:35:15     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2024-09-20 09:35:15     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2024-09-20 09:35:15     at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
2024-09-20 09:35:15     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
2024-09-20 09:35:15     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
2024-09-20 09:35:15     at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-09-20 09:35:15 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
2024-09-20 09:35:15     at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
2024-09-20 09:35:15     at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
2024-09-20 09:35:15     at jdk.internal.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
2024-09-20 09:35:15     at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
2024-09-20 09:35:15     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
2024-09-20 09:35:15     at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
2024-09-20 09:35:15     at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive(Actor.scala:537)
2024-09-20 09:35:15     at akka.actor.Actor.aroundReceive$(Actor.scala:535)
2024-09-20 09:35:15     at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
2024-09-20 09:35:15     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
2024-09-20 09:35:15     at akka.actor.ActorCell.invoke(ActorCell.scala:548)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.run(Mailbox.scala:231)
2024-09-20 09:35:15     at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
2024-09-20 09:35:15     ... 5 common frames omitted
2024-09-20 09:35:15 Caused by: org.apache.beam.sdk.util.UserCodeException: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
2024-09-20 09:35:15     at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
2024-09-20 09:35:15     at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
2024-09-20 09:35:15     at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
2024-09-20 09:35:15     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
2024-09-20 09:35:15     at java.base/java.lang.Thread.run(Thread.java:833)
2024-09-20 09:35:15 Caused by: ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: expected datetime: found "2024-08-06T20:35:06.548462"
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
2024-09-20 09:35:15     at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
2024-09-20 09:35:15     at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:317)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:402)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:585)
2024-09-20 09:35:15     at ca.uhn.fhir.rest.client.impl.GenericClient$TransactionExecutable.execute(GenericClient.java:2382)
2024-09-20 09:35:15     at com.google.fhir.analytics.FhirStoreUtil.uploadBundle(FhirStoreUtil.java:146)
2024-09-20 09:35:15     at com.google.fhir.analytics.GcpStoreUtil.uploadBundle(GcpStoreUtil.java:81)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:287)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:267)
2024-09-20 09:35:15     at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:147)
2024-09-20 09:35:15 02:35:15.692 [Thread-3] INFO  c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:708 - Total time taken for the pipelines = 7 secs
@bashir2
Copy link
Collaborator

bashir2 commented Sep 19, 2024

Can you please provide a sample config (or command line if you are running the pipelines directly and not through the controller) to reproduce this? Also please provide the full stack-trace of the errors if possible.

@muhammad-levi
Copy link
Collaborator Author

muhammad-levi commented Sep 20, 2024 •
edited
Loading

@bashir2 After further investigation yesterday, it seems that our data values were not compliant with what FHIR spec described on dateTime as our datetime values were lacking time zone.

From the spec:

If hours and minutes are specified, a time zone SHALL be populated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bashir2 @muhammad-levi