东杰书屋 2020-01-28
apiVersion: v1 kind: PersistentVolume metadata: name: es-data-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce storageClassName: nfs nfs: server: 192.168.0.250 path: /var/nfs/es-data --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: es-data-pvc spec: accessModes: - ReadWriteOnce storageClassName: "nfs" resources: requests: storage: 10Gi --- apiVersion: apps/v1 kind: Deployment metadata: name: es-deployment spec: replicas: 1 selector: matchLabels: app: single-es template: metadata: labels: app: single-es spec: initContainers: - name: init-sysctl image: busybox imagePullPolicy: IfNotPresent command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true containers: - name: single-es image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2 ports: - containerPort: 9200 name: http - containerPort: 9300 name: transport env: - name: discovery.type value: single-node volumeMounts: - mountPath: /usr/share/elasticsearch/data name: es-data volumes: - name: es-data persistentVolumeClaim: claimName: es-data-pvc --- apiVersion: v1 kind: Service metadata: name: single-es-svc spec: selector: app: single-es type: NodePort ports: - name: http port: 9200 targetPort: 9200 nodePort: 30092 --- apiVersion: apps/v1 kind: Deployment metadata: name: kibana-deployment spec: replicas: 1 selector: matchLabels: app: kibana template: metadata: labels: app: kibana spec: containers: - name: kibana image: docker.elastic.co/kibana/kibana:7.5.2 ports: - name: http containerPort: 5601 env: - name: ELASTICSEARCH_HOSTS value: http://single-es-svc:9200 --- apiVersion: v1 kind: Service metadata: name: kibana-svc spec: selector: app: kibana type: NodePort ports: - name: http port: 5601 targetPort: 5601 nodePort: 30561
参考网站:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
修改 fluentd-es-configmap.yaml 中的
output.conf: |- <match **> @id elasticsearch @type elasticsearch @log_level info type_name _doc include_tag_key true host elasticsearch-logging port 9200 logstash_format true <buffer> @type file path /var/log/fluentd-buffers/kubernetes.system.buffer flush_mode interval retry_type exponential_backoff flush_thread_count 2 flush_interval 5s retry_forever retry_max_interval 30 chunk_limit_size 2M total_limit_size 500M overflow_action block </buffer> </match>
match 中的es的host改为部署的es的服务名 elasticsearch-logging -> single-es-svc.default // {服务名.命名空间},如果 es 与 fluentd 在一个命名空间下可省略。
另外一部分,则需要先做聚类、分类处理,将聚合出的分类结果存入ES集群的聚类索引中。数据处理层的聚合结果存入ES中的指定索引,同时将每个聚合主题相关的数据存入每个document下面的某个field下。