Absolute imports for proto files in the same directory not working #116

rbroggi · 2022-06-20T06:37:47Z

Absolute imports for `proto` files in the same directory not working

In case two or more proto files are in the same package and there is a dependency between them, the go code generation will not work correctly in case the import statement is provided in absolute path. It works correctly if the import statement is relative.

Example:

In a project with the following directory structure:

❯ tree
.
├── proto
│   └── account
│       ├── models.proto
│       └── services.proto
└── readme.md

where the files have the following content:

models.proto

syntax = "proto3";

package services;

option go_package = "github.com/proto/services";

message Balance {
  int64 units = 1;
  string currency = 2;
}

services.proto

syntax = "proto3";

package services;

option go_package = "github.com/proto/services";

import "account/models.proto";

service BalanceSvc {
  rpc GetBalance(GetBalanceRequest) returns (GetBalanceResponse);
}

message GetBalanceRequest {
  int64 account_id = 1;
}

message GetBalanceResponse {
  Balance balance = 1;
}

In this example we have a services.proto which defines the grpc services and a models.proto which defines the models for the service. services.proto imports models.proto and both are in the same package and define same go_package. Notice how the --imports flag allows gripmock to find the models.proto but it seems like it is found twice and therefore issues an error. The command below is issued from the project root:

❯ docker run -p 4770:4770 -p 4771:4771  -v "$PWD/proto:/proto" tkpd/gripmock --imports="/protobuf,/proto" /proto/account/services.proto /proto/account/models.proto
Starting GripMock
Serving stub admin on http://:4771
models.proto:10:9: "services.Balance.units" is already defined in file "account/models.proto".
models.proto:11:10: "services.Balance.currency" is already defined in file "account/models.proto".
models.proto:9:9: "services.Balance" is already defined in file "account/models.proto".
2022/06/20 06:19:22 Fail on protoc exit status 1

Interestingly enough, changing the import statement in services.proto from import "account/models.proto"; to import "models.proto"; works.

Attached you can find a zip file containing the project to reproduce the issue.
absolute-import-path-bug.zip

The text was updated successfully, but these errors were encountered:

jekiapp · 2022-06-28T02:33:58Z

when you're not using gripmock, how you pass the argument into protoc?
since above error is directly from protoc.

daymansiege · 2022-07-19T15:39:07Z

I have similar issue. Service references shared models (shared between multiple services) and I add them as to imports.
Doesn't matter if they are in the same folder (/protos) or multiple import paths (/protos,/common), gripmock seems to find them and even read package name but fails to load them, see sample output (just replaced companu name with XXX)

grpc server pid: 65
/go/src/grpc/server.go:19:2: cannot find package "XXX/commonprotos/rpc/EmptyResponse" in any of:
        /usr/local/go/src/XXX/commonprotos/rpc/EmptyResponse (from $GOROOT)
        /go/src/XXX/commonprotos/rpc/EmptyResponse (from $GOPATH)
2022/07/19 14:24:49 exit status 1

With v1.11-beta message is a bit different (tries to load definition from web) but seems the problem is the same:

2022/07/19 14:25:35 error on fixGoPackage exit status 1
go: finding module for package XXX/commonprotos/rpc/EmptyResponse
/go/src/grpc/server.go:19:2: cannot find module providing package XXX/commonprotos/rpc/EmptyResponse: unrecognized import path "XXX/commonprotos/rpc/EmptyResponse": reading https://XXX/commonprotos/rpc/EmptyResponse?go-get=1: 404 Not Found
2022/07/19 14:25:38 exit status 1

And if I try to add common proto to list of protos explicitly (i.e. '/common/XXX/api/rpc/empty_response.proto' '/proto/AccountService.proto'), it fails like in author's case that type is already defined:

2022/07/19 15:24:39 error on fixGoPackage exit status 1
XXX/api/rpc/empty_response.proto:16:39: "XXX.API.CommonProtos.RPC.EmptyResponse.status" is already defined in file "empty_response.proto".
XXX/api/rpc/empty_response.proto:15:9: "XXX.API.CommonProtos.RPC.EmptyResponse" is already defined in file "empty_response.proto".
AccountService.proto:4:1: Import "XXX/api/rpc/empty_response.proto" was not found or had errors.
AccountService.proto:53:73: "XXX.API.CommonProtos.RPC.EmptyResponse" seems to be defined in "empty_response.proto", which is not imported by "AccountService.proto".  To use it here, please add the necessary import.
AccountService.proto:56:69: "XXX.API.CommonProtos.RPC.EmptyResponse" seems to be defined in "empty_response.proto", which is not imported by "AccountService.proto".  To use it here, please add the necessary import.
2022/07/19 15:24:39 Fail on protoc exit status 1

To generate real service code we use dotnet Grpc.Tools, which also uses protoc under the hood, and from logs I see it simly uses multiple --proto_path directives w/o any issue, so it seems like some Go specific issue, but I'm not sure how to diagnose :)

daymansiege · 2022-07-19T18:40:53Z

Here is simplified example of the issue (see code attached test.zip)
Something simple that uses well known google type, works fine.

version: "3.7"
services:
  hello-service:
    image: tkpd/gripmock:latest
    ports:
      - '4770:4770'
      - '4771:4771'
    command:
      - '--imports=/protobuf,/proto'
      - '/proto/hello-service.proto'
    volumes:     
      - './../TestCompany.Service/proto:/proto:ro'

Starting GripMock
Serving stub admin on http://:4771
grpc server pid: 60
Serving gRPC on tcp://:4770

If using some imports from external folder, getting error:

version: "3.7"
services:
  job-service:
    image: tkpd/gripmock:latest
    ports:
      - '4770:4770'
      - '4771:4771'
    command:
      - '--imports=/protobuf,/common,/proto'
      - '/proto/job-service.proto'
    volumes:
      - './../TestCompany.Common/proto:/common:ro'      
      - './../TestCompany.Service/proto:/proto:ro'

Starting GripMock
Serving stub admin on http://:4771
grpc server pid: 65
/go/src/test.company/services/test-service/job-service.pb.go:19:2: cannot find package "test.company/common/rpc/status" in any of:
        /usr/local/go/src/test.company/common/rpc/status (from $GOROOT)
        /go/src/test.company/common/rpc/status (from $GOPATH)
2022/07/19 18:31:17 exit status 1

And to reproduce author's behavior, add dependency to compile explicitly and it fails with duplicated definitions:

version: "3.7"
services:
  job-service:
    image: tkpd/gripmock:latest
    ports:
      - '4770:4770'
      - '4771:4771'
    command:
      - '--imports=/protobuf,/common,/proto'
      - '/common/tc/rpc/status.proto'
      - '/proto/job-service.proto'
    volumes:
      - './../TestCompany.Common/proto:/common:ro'      
      - './../TestCompany.Service/proto:/proto:ro'

Starting GripMock
Serving stub admin on http://:4771
tc/rpc/status.proto:11:31: "TestCompany.Common.Rpc.Status.code" is already defined in file "status.proto".
tc/rpc/status.proto:12:10: "TestCompany.Common.Rpc.Status.message" is already defined in file "status.proto".
tc/rpc/status.proto:13:32: "TestCompany.Common.Rpc.Status.details" is already defined in file "status.proto".
tc/rpc/status.proto:14:10: "TestCompany.Common.Rpc.Status.trace_id" is already defined in file "status.proto".
tc/rpc/status.proto:10:9: "TestCompany.Common.Rpc.Status" is already defined in file "status.proto".
job-service.proto:4:1: Import "tc/rpc/status.proto" was not found or had errors.
job-service.proto:19:3: "TestCompany.Common.Rpc.Status" seems to be defined in "status.proto", which is not imported by "job-service.proto".  To use it here, please add the necessary import.
2022/07/20 06:38:15 Fail on protoc exit status 1

sdontireddy · 2022-09-22T16:44:37Z

+1 , we also have same issue.
As a temp solution , we had to put all the required protos under the same folder and updated the imports accordingly.

cezarypiatekGC · 2023-05-12T12:04:49Z

I solved the issue by specifying all imported files explicitly along the main proto file. It would be helpful if gripmock could accept path pattern (glob) and expand it automatically like:

docker run -p 4770:4770 -p 4771:4771 -v c:\temp\prototest:/proto tkpd/gripmock /proto/**/*.proto

https://stackoverflow.com/questions/26809484/how-to-use-double-star-glob-in-go

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Absolute imports for proto files in the same directory not working #116

Absolute imports for proto files in the same directory not working #116

rbroggi commented Jun 20, 2022

jekiapp commented Jun 28, 2022

daymansiege commented Jul 19, 2022

daymansiege commented Jul 19, 2022 •

edited

Loading

sdontireddy commented Sep 22, 2022

cezarypiatekGC commented May 12, 2023 •

edited

Loading

Absolute imports for proto files in the same directory not working #116

Absolute imports for proto files in the same directory not working #116

Comments

rbroggi commented Jun 20, 2022

Absolute imports for proto files in the same directory not working

jekiapp commented Jun 28, 2022

daymansiege commented Jul 19, 2022

daymansiege commented Jul 19, 2022 • edited Loading

sdontireddy commented Sep 22, 2022

cezarypiatekGC commented May 12, 2023 • edited Loading

Absolute imports for `proto` files in the same directory not working

daymansiege commented Jul 19, 2022 •

edited

Loading

cezarypiatekGC commented May 12, 2023 •

edited

Loading