Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi es-instances support #1225

Closed
wants to merge 3 commits into from
Closed

multi es-instances support #1225

wants to merge 3 commits into from

Conversation

clarkchen
Copy link

I take advantages of Elasticsearch, make host param as a list.

Host Config Change to
"host1:port1, host2:port " can handle multi es-instances

while the original port and host can also work well here

@clarkchen
Copy link
Author

hello?

Copy link
Member

@Qmando Qmando left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation?

if "es_host" in conf:
es_host = os.environ.get('ES_HOST', conf['es_host'])

if "es_host" in conf:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

es_port

Though, this also isn't necessary.

parsed_conf['es_url_prefix'] = ''
parsed_conf['es_conn_timeout'] = conf.get('es_conn_timeout', 20)
parsed_conf['send_get_body_as'] = conf.get('es_send_get_body_as', 'GET')

if "es_host" in conf:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check isn't necessary, it's always there.

@@ -344,6 +347,18 @@ def build_es_conn_config(conf):
return parsed_conf


def parse_host(host, port="9200"):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not accept a list instead of a comma separated string?

@YoavBZ
Copy link

YoavBZ commented Sep 7, 2017

Nice! I've just started working on a similar feature :)
In my case, I wanted to run rules on few different hosts and didn't wanted to duplicates my rules, so I added a list of objects to the rule .yaml file that includes the ES hosts, ports and names, so the rule will run separately on each host.
I also appended the environment name to the rule name, i.e "Rule name [Env name]".

How this one is going to work?

@Qmando
Copy link
Member

Qmando commented Sep 7, 2017

I guess the motivation here is that you could replace a load balancer by having a pool of servers to roundrobin requests to.

@YoavBZ
Copy link

YoavBZ commented Sep 7, 2017

I see.. My idea was more for multiple environments support

@Qmando
Copy link
Member

Qmando commented Sep 7, 2017

So you implemented this feature to avoid duplicating YAML files, is that correct?

You can import bits and pieces of rules, so you could specify two files with just a es_host, name and import parameter.

@YoavBZ
Copy link

YoavBZ commented Sep 7, 2017

Yes, I wanted to avoid duplicating cause I have one rule with 2/3 ES hosts I want to run it on.
So instead of creating few duplicate rules but with different es_hosts I created one rule with few es_hosts in it

@YoavBZ
Copy link

YoavBZ commented Sep 7, 2017

*Not one rule actually, I need to run all my rules on few es_hosts

@sathishdsgithub
Copy link

@YoavBZ @Qmando @clarkchen

Can we make use of the above modified util.py for multi ES instance? I tried with below format but it did not work.
Please, can someone suggest me the right method to mention the list of ES host entry with port?

es_host: "192.168.96.157:9200, 192.168.96.141:9200"

@nsano-rururu
Copy link
Contributor

Can you resolve conflicts please :)

@clarkchen
Copy link
Author

clarkchen commented Jan 30, 2021

Can you resolve conflicts please :)

sorry, I have lost my origin repo, so I start a new pull request, where you can review code there.

thx~

@clarkchen clarkchen closed this Jan 30, 2021
@clarkchen
Copy link
Author

@sathishdsgithub

the ElasticSearchClient init should change to hosts
image

I add a test case which connect with a docker-composed elastic cluste

you can find the test case here

hope it will work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants