[ Web Proxy ]
URL:
Viewing: https://raw.githubusercontent.com/We2one/Notes/master/python-basic-notes/python-days/Day_47.md [Back]  [Original]

### Tornado #### Elasticsearch ##### Elasticsearch + Elasticsearch . + Elasticsearch , . + Elasticsearch , Apache Lucene . + Lucene ,, Java Lucene + Elasticsearch Java , Lucene ,, Lucene , RESTful API + Elasticsearch : + , + , + , PB + Elasticsearch , RESTful API , Web #### Elasticsearch ##### Centos7 1. : `wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.1.tar.gz` 2. home : `tar -zxvf elasticsearch-6.3.1.tar.gz` 3. ES 1. `adduser esuser` : 2. `cd /home` : home 3. `mkdir -p esdata/data` : 4. `mkdir -p esdata/log` : 5. `chown -R esuser elasticsearch-6.3.1` : 6. `chown -R esuser esdata` : : 7. : `systemctl stop firewawlld` 8. : `systemctl status firewalld` 4. ES 1. , , ``` cd /home/elaticsearch-6.3.1/config/ cp elasticsearch.yml elasticsearch.yml.bak vi elasticsearch.yml ``` 2. ``` # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #cluster.name: my-application cluster.name: my-application # # ``` 3. ``` # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: node-1 node.name: node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # ``` 4. es ``` # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data path.data: /home/esdata/data # # Path to log files: # #path.logs: /path/to/logs path.logs: /home/esdata/log # ``` 5. ``` # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # ``` 6. ip ``` # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 network.host: 0.0.0.0 # # Set a custom port for HTTP: # #http.port: 9200 http.port: 9200 # # For more information, consult the network module documentation. # ``` 7. ``` # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # ``` 8. requires_name ``` # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 gateway.recover_after_nodes: 1 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true ``` 5. 1. `vim /etc/security/limits.conf` : ``` # End of file esuser hard nofile 65536 esuser soft nofile 65536 esuser soft memlock unlimited esuser hard memlock unlimited ``` 2. `vi /etc/sysctl.conf` ``` # sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. # # Vendors settings live in /usr/lib/sysctl.d/. # To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). vm.max_map_count=262144 ``` 3. : 1. `sysctl -p` 2. `visudo` : esuser ``` ## The COMMANDS section may have other options added to it. ## ## Allow root to run any commands anywhere root ALL=(ALL) ALL esuser ALL=(ALL) ALL ``` 4. `vim /etc/security/limits.d/20-nproc.conf` ``` # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz #432903 for reasoning. * soft nproc 4096 root soft nproc unlimited * soft nproc 204800 * hard nproc 204800 ``` 5. `vim /etc/security/limits.d/def.conf` : , ``` * soft nofile 204800 * hard nofile 204800 ~ ~ ``` 6. 1.8 java (`yum install java -y`) + elasticsearch-6.3.1 : `cd /home/elasticsearch-6.3.1/bin/` + esuser : `su esuser` + : `./elasticsearch` + ip 9200 ##### Postman 1. Postman + Postman HTTP Chrome + B/S , (Firebug) + Postman CSSHTML, HTTP 2. [Postman (Google )](https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop/related?hl=zh-CN) Postman.png [Postman.png] 3. 1. : ,, 2. : 3. : , ##### ES 1. 1. Es | | database | table | rows | Columns | | ------------- | -------- | ----- | -------- | ------- | | Elasticsearch | index | type | document | fields | 2. (), () 3. Es 2. (index) 1. , put 1. postman_index01.png [postman_index01.png] postman_index02.png [postman_index02.png] 2. ,, | | | | ---------- | ----------------- | | carlist | , | | mappings | , | | car | , | | properties | | | c_name | , | | type | | | text | | 3. Es | | | | ------- | ------------------------------------------------------------ | | string | | | long | 64 , | | integer | 32 , | | short | 16 , | | byte | 8 , | | double | 64 , | | float | 32 , | | date | , | | Boolean | | | Binary | | | Array | | | Object | json () | | nested | json | | text | ,, Analyzer
, fielddata true | | Keyword | ID,
Filter ,Sorting Aggregations | 2. : `ip/_/indices/` + postman_getindex01.png [postman_getindex01.png] + `yellow open hello rs7mwZXPQL6d-bKUy_DKng 5 1 0 0 1.2kb 1.2kb` + yellow : , ,, + open : , close + index_name : 3. : `ip:9200//` + postman_getindex02.png [postman_getindex02.png] 4. : `ip:9200//_search` + postman_getindex03.png [postman_getindex03.png] 5. : DELETE + postman_delete.png [postman_delete.png] 3. (document post ) : `ip//` + postman_post01.png [postman_post01.png] 4. (document get ) : `ip///_search` + postman_post02.png [postman_post02.png] #### ES ##### ES postman_esmodel.png [postman_esmodel.png] ##### Python ES 1. elasticsearch : `pip install elasticsearch -i http://pypi.tuna.tsinghua.edu.cn/simple` 2. ```python import random from elasticsearch import Elasticsearch es = Elasticsearch("10.10.123.131", timeout=360) # , # 10000 c_names = """ """.split("\n") c_citys = "\ \ \ ".split("") for i in range(10000): data = { "c_name": random.choice(c_names), "c_date": "%s-%s-%s" % ( random.randint(1983, 2020), random.randint(1, 12), random.randint(1, 28) ), "c_mileage": random.randint(1, 10000), "c_city": random.choice(c_citys), "c_price": random.randint(8, 100), "c_sale": random.randint(1, 3), "c_service": random.randint(30, 100) } # res = es.index(index="carlist", doc_type="car", body=data) # print(res) print('....') ``` #### ES ##### (Python postman GET ) 1. ```json { "query": { "": { "": "" } } } ``` 2. **term** ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "term": { "c_name": "" } } } result = es.search(index="carlist", doc_type="car", body=body) print(result) print(result["hits"]["hits"]) # ``` 3. **terms** ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "terms": { "c_name": ["", ""] } } } result = es.search(index="carlist", doc_type="car", body=body) print(result) print(result["hits"]["hits"]) # ``` 4. **match_all** ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "match_all":{ } } } result = es.search(index="carlist", doc_type="car", body=body) print(result) print(len(result["hits"]["hits"])) # 10 10 ``` 5. : **from**: , **size**: ( from ) ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "match_all":{ } }, "from": 0, "size": 100 } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) # 100 print(result["hits"]["hits"]) ``` 6. **sort** 1. `"sort: [ '' ]"` ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "match_all":{ }, "sort": [ "c_date" # ], "from": 0, "size": 100 } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 2. : ``"sort: { '': { 'order': 'desc' } }"`` ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "match_all":{ }, }, "sort": { "c_date": { "order": "desc" } }, "from": 0, "size": 100 } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 7. : **match** ,****, ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { "query": { "match": { "c_name": "" } } } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 8. **bool** 1. ```json { "query": { "bool": { "": [ { "1": { "1": "1" } }, { "2": { "2": "2" } }, { "3": { "3": "3" } }, ] } } } ``` 2. **must** : , ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { "query": { "bool": { "must": [ { "term": { "c_name": "" } }, { "term": { "c_price": 18 } } ] } } } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 3. **should** : , ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { "query": { "bool": { "should": [ { "term": { "c_name": "" } }, { "term": { "c_price": 18 } } ] } } } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 4. **must_not** : , ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { "query": { "bool": { "must_not": [ { "term": { "c_name": "" } }, ] } } } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 5. must should : must shoud ,should bool must ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) # 18 body = { "query": { "bool": { "must": [ { "term": { "c_price": 18 } }, { "bool": { "should": [ { "term": { "c_name": "" } }, { "term": { "c_name": "" } } ] } } ] } } } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 9. **range** 1. | | | | ---- | -------- | | lt | | | lte | | | gt | | | gte | | 2. ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # 10 -15 "query": { "range": { "c_price": { "lt": 15, "gt": 10 } } }, "from": 0, "size": 100, } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["hits"]["hits"]) ``` 10. **aggs** 1. | | | | ------------ | -------- | | avg | | | sum | | | max | | | min | | 2. ``` { "query": { "": { "": "" } }, "aggs": { "": { "": { "field": "" } } } } ``` 3. ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) body = { # "query": { "term": { "c_name": "" } }, "aggs": { "price_avg": { "avg": { "field": "c_price" } } } } result = es.search(index="carlist", doc_type="car", body=body) print(result) print(len(result["hits"]["hits"])) print(result["aggregations"]["price_avg"]["value"]) # ``` 11. ```python from elasticsearch import Elasticsearch es = Elasticsearch(hosts="10.10.123.131", timeout=360) # body = { "query": { "match_all": { } }, "aggs": { "case_sum": { "terms": { "field": "c_name" }, "aggs": { "price_avg": { "avg": { "field": "c_price" } } } } } } result = es.search(index="carlist", doc_type="car", body=body) # print(result) print(len(result["hits"]["hits"])) print(result["aggregations"]["case_sum"]) print(result["aggregations"]["case_sum"]["buckets"]) ```
Web Proxy Viewer  |  New URL  |  Original Page