环境

Ubuntu 22.04 * 3
Docker version 27.2.1, build 9e34c9b
Docker Compose version v2.20.2
Graylog 6.0.6
Opensearch 2.15.0
MongoDB 6.0.17

目的

多主机集群实例，多服务的实际生产业务。非单主机集群。
请注意更改主机名，以下为：ubuntu01，t01-ubuntu01，t02-ubuntu02。
ubuntu01：graylog1(9000/tcp,11514/tcp+udp,12201/tcp+udp,5044/tcp), opensearch1(9200/tcp), mongodb1(27017/tcp)
t01-ubuntu01：graylog2(9000/tcp,11514/tcp+udp,12201/tcp+udp,5044/tcp), opensearch1(9200/tcp), mongodb1(27017/tcp)
t02-ubuntu02：graylog3(9000/tcp,11514/tcp+udp,12201/tcp+udp,5044/tcp), opensearch1(9200/tcp), mongodb1(27017/tcp)
MongoDB采用副本集，graylog1 固定为 Leader。
所有容器均固定在对应主机上，不可漂移。
访问主机端口即访问到对应业务（即不使用 Docker Swarm 自带的路由功能来进行负载均衡）。
若要启用漂移，请使用路由负载均衡模式，并修改重复的冲突端口，并修改对应业务参数设置；例如 MongoDB 默认参数是根据 1 台宿主机 1 台 MongoDB 而生成的，当多台长期运行在一台上时，请增加资源限制，或设置相关参数。
Graylog Web 默认密码：Qwer.123（更改：echo -n 密码 | shasum -a 256的值，替换 ）
Opensearch 密码：P@ssw0rd.yudelei.com
通信未加密，请注意权限，若无必要，可不发布（注释） opensearch 与 mongodb 相关端口。
使用主机卷（Bind Mount 模式），未使用命名卷模式（Volume），若中大型生产业务建议直接上存储集群，分布式存储：GlusterFS，S3等。

操作

部署Docker Swarm 过程忽略。
前期设置，三台主机均需要操作。

apt install nfs-common -y
mkdir /data-nfs && mount -t nfs 192.168.120.52:/DockerData /data-nfs
# 若要启动即挂载，执行
echo "192.168.120.52:/DockerData    /data-nfs   nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0" >> /etc/fstab

vim /etc/sysctl.conf
# 新增或修改 vm.max_map_count=262144
sysctl -p

创建共享目录、相关用户并授权

mkdir -p /data-nfs/graylog/graylog1-data /data-nfs/graylog/graylog2-data /data-nfs/graylog/graylog3-data \
/data-nfs/graylog/graylog1-journal /data-nfs/graylog/graylog2-journal /data-nfs/graylog/graylog3-journal \
/data-nfs/graylog/opensearch1-data /data-nfs/graylog/opensearch2-data /data-nfs/graylog/opensearch3-data \
/data-nfs/graylog/mongodb1-data /data-nfs/graylog/mongodb2-data /data-nfs/graylog/mongodb3-data

# 若本机只有 root 用户
useradd -u 1000 opensearch
# 若本机除了 root 用户，还有普通用户
useradd -u 1100 graylog
chown -R 1100:root /data-nfs/graylog/graylog*
chown -R 1000:root /data-nfs/graylog/opensearch*

#删除命令
rm -rf /data-nfs/graylog/graylog1-data /data-nfs/graylog/graylog2-data /data-nfs/graylog/graylog3-data \
/data-nfs/graylog/graylog1-journal /data-nfs/graylog/graylog2-journal /data-nfs/graylog/graylog3-journal \
/data-nfs/graylog/opensearch1-data /data-nfs/graylog/opensearch2-data /data-nfs/graylog/opensearch3-data \
/data-nfs/graylog/mongodb1-data /data-nfs/graylog/mongodb2-data /data-nfs/graylog/mongodb3-data

Stack Compose File

格式规范 Compose V1 格式3（旧版 Compose，非 Compose V2），通常为 compose.yml，个人习惯为 stack-compose.yml。
由于主机由其他日志程序占用了 1514，则将 Graylog 中相关端口监听设置为 11514.
将以下文件保存为 stack-compose.yml
部署：docker stack deploy -c stack-compose.yml gl-stack
删除：docker stack rm gl-stack

version: "3.8"
services:
  mongodb1:
    image: mongo:6.0.17
    hostname: "mongodb1"
    command: ["--replSet", "rs0", "--bind_ip_all", "--port", "27017"]
    ports:
      - target: 27017
        published: 27017
        mode: host
    healthcheck:
      test: echo "try { rs.status() } catch (err) { rs.initiate({_id:'rs0',members:[{_id:0,host:'mongodb1:27017',priority:1},{_id:1,host:'mongodb2:27017',priority:0.5},{_id:2,host:'mongodb3:27017',priority:0.5}]}) }" | mongosh --port 27017 --quiet
      interval: 5s
      timeout: 30s
      start_period: 0s
      retries: 30
    volumes:
      - type: bind
        source: /data-nfs/graylog/mongodb1-data
        target: /data
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == ubuntu01
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  mongodb2:
    image: mongo:6.0.17
    hostname: "mongodb2"
    command: ["--replSet", "rs0", "--bind_ip_all", "--port", "27017"]
    ports:
      - target: 27017
        published: 27017
        mode: host
    volumes:
      - type: bind
        source: /data-nfs/graylog/mongodb2-data
        target: /data
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == t01-ubuntu
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  mongodb3:
    image: mongo:6.0.17
    hostname: "mongodb3"
    command: ["--replSet", "rs0", "--bind_ip_all", "--port", "27017"]
    ports:
      - target: 27017
        published: 27017
        mode: host
    volumes:
      - type: bind
        source: /data-nfs/graylog/mongodb3-data
        target: /data
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == t02-ubuntu
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  opensearch1:
    image: opensearchproject/opensearch:2.15.0
    hostname: "opensearch1"
    depends_on:
      - "mongodb1"
      - "mongodb2"
      - "mongodb3"
    ports:
      - target: 9200
        published: 9200
        mode: host
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      - "OPENSEARCH_HEAP_SIZE=1g"
      - "node.name=opensearch1"
      - "cluster.name=opensearch-graylog-cluster"
      - "discovery.seed_hosts=opensearch2,opensearch3"
      - "cluster.initial_master_nodes=opensearch1,opensearch2,opensearch3"
      - "bootstrap.memory_lock=true"
      - "action.auto_create_index=false"
      - "plugins.security.ssl.http.enabled=false"
      - "plugins.security.disabled=true"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Qwer.123,m,."
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - type: bind
        source: /data-nfs/graylog/opensearch1-data
        target: /usr/share/opensearch/data
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == ubuntu01
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  opensearch2:
    image: opensearchproject/opensearch:2.15.0
    hostname: "opensearch2"
    depends_on:
      - "mongodb1"
      - "mongodb2"
      - "mongodb3"
    ports:
      - target: 9200
        published: 9200
        mode: host
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      - "OPENSEARCH_HEAP_SIZE=1g"
      - "node.name=opensearch2"
      - "cluster.name=opensearch-graylog-cluster"
      - "discovery.seed_hosts=opensearch1,opensearch3"
      - "cluster.initial_master_nodes=opensearch1,opensearch2,opensearch3"
      - "bootstrap.memory_lock=true"
      - "action.auto_create_index=false"
      - "plugins.security.ssl.http.enabled=false"
      - "plugins.security.disabled=true"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Qwer.123,m,."
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - type: bind
        source: /data-nfs/graylog/opensearch2-data
        target: /usr/share/opensearch/data
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == t01-ubuntu
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  opensearch3:
    image: opensearchproject/opensearch:2.15.0
    hostname: "opensearch3"
    depends_on:
      - "mongodb1"
      - "mongodb2"
      - "mongodb3"
    ports:
      - target: 9200
        published: 9200
        mode: host
    environment:
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      - "OPENSEARCH_HEAP_SIZE=1g"
      - "node.name=opensearch3"
      - "cluster.name=opensearch-graylog-cluster"
      - "discovery.seed_hosts=opensearch1,opensearch2"
      - "cluster.initial_master_nodes=opensearch1,opensearch2,opensearch3"
      - "bootstrap.memory_lock=true"
      - "action.auto_create_index=false"
      - "plugins.security.ssl.http.enabled=false"
      - "plugins.security.disabled=true"
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Qwer.123,m,."
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - type: bind
        source: /data-nfs/graylog/opensearch3-data
        target: /usr/share/opensearch/data
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == t02-ubuntu
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  graylog1:
    image: graylog/graylog:6.0.6
    hostname: "graylog1"
    depends_on:
      - "mongodb1"
      - "mongodb2"
      - "mongodb3"
      - "opensearch1"
      - "opensearch2"
      - "opensearch3"
    entrypoint: "/docker-entrypoint.sh"
    environment:
      GRAYLOG_IS_LEADER: "true"
      #GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/data/node-id"
      GRAYLOG_PASSWORD_SECRET: "DQPDrSXICSrz2BG7WScmxtUCHguATRbDoB43PcwoOxMZCInQyrq7Kb4sMVSpSEHTTGZgwmYrunzWToLhT9CdeWsrEKQByMeV"
      GRAYLOG_ROOT_PASSWORD_SHA2: "23d53009e0a2dd91eb7bf459698abc50bf8f5ff0ef1d17442c9b8a51cebcd854"
      GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
      GRAYLOG_HTTP_PUBLISH_URI: "http://graylog1:9000/"
      #GRAYLOG_HTTP_EXTERNAL_URI: "http://log1.it.yudelei.com:9000/"
      GRAYLOG_ELASTICSEARCH_HOSTS: "http://opensearch1:9200,http://opensearch2:9200,http://opensearch3:9200"
      GRAYLOG_MONGODB_URI: "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/graylog"
    ports:
      - target: 9000
        published: 9000
        mode: host
      - target: 1514 # Syslog TCP
        published: 11514
        mode: host
      - target: 1514 # Syslog UDP
        published: 11514
        protocol: udp 
        mode: host
      - target: 12201 # GELF TCP
        published: 12201
        mode: host
      - target: 12201 # GELF UDP
        published: 12201
        protocol: udp
        mode: host
      - target: 5044 # Logstash
        published: 5044
        mode: host
    volumes:
      - type: bind
        source: /data-nfs/graylog/graylog1-data
        target: /usr/share/graylog/data/data
      - type: bind
        source: /data-nfs/graylog/graylog1-journal
        target: /usr/share/graylog/data/journal
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == ubuntu01
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  graylog2:
    image: graylog/graylog:6.0.6
    hostname: "graylog2"
    depends_on:
      - "mongodb1"
      - "mongodb2"
      - "mongodb3"
      - "opensearch1"
      - "opensearch2"
      - "opensearch3"
    entrypoint: "/docker-entrypoint.sh"
    environment:
      GRAYLOG_IS_LEADER: "false"
      #GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/data/node-id"
      GRAYLOG_PASSWORD_SECRET: "DQPDrSXICSrz2BG7WScmxtUCHguATRbDoB43PcwoOxMZCInQyrq7Kb4sMVSpSEHTTGZgwmYrunzWToLhT9CdeWsrEKQByMeV"
      GRAYLOG_ROOT_PASSWORD_SHA2: "23d53009e0a2dd91eb7bf459698abc50bf8f5ff0ef1d17442c9b8a51cebcd854"
      GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
      GRAYLOG_HTTP_PUBLISH_URI: "http://graylog2:9000/"
      #GRAYLOG_HTTP_EXTERNAL_URI: "http://log2.it.yudelei.com:9000/"
      GRAYLOG_ELASTICSEARCH_HOSTS: "http://opensearch1:9200,http://opensearch2:9200,http://opensearch3:9200"
      GRAYLOG_MONGODB_URI: "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/graylog"
    ports:
      - target: 9000
        published: 9000
        mode: host
      - target: 1514 # Syslog TCP
        published: 11514
        mode: host
      - target: 1514 # Syslog UDP
        published: 11514
        protocol: udp 
        mode: host
      - target: 12201 # GELF TCP
        published: 12201
        mode: host
      - target: 12201 # GELF UDP
        published: 12201
        protocol: udp
        mode: host
      - target: 5044 # Logstash
        published: 5044
        mode: host
    volumes:
      - type: bind
        source: /data-nfs/graylog/graylog2-data
        target: /usr/share/graylog/data/data
      - type: bind
        source: /data-nfs/graylog/graylog2-journal
        target: /usr/share/graylog/data/journal
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == t01-ubuntu
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

  graylog3:
    image: graylog/graylog:6.0.6
    hostname: "graylog3"
    depends_on:
      - "mongodb1"
      - "mongodb2"
      - "mongodb3"
      - "opensearch1"
      - "opensearch2"
      - "opensearch3"
    entrypoint: "/docker-entrypoint.sh"
    environment:
      GRAYLOG_IS_LEADER: "false"
      #GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/data/node-id"
      GRAYLOG_PASSWORD_SECRET: "DQPDrSXICSrz2BG7WScmxtUCHguATRbDoB43PcwoOxMZCInQyrq7Kb4sMVSpSEHTTGZgwmYrunzWToLhT9CdeWsrEKQByMeV"
      GRAYLOG_ROOT_PASSWORD_SHA2: "23d53009e0a2dd91eb7bf459698abc50bf8f5ff0ef1d17442c9b8a51cebcd854"
      GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
      GRAYLOG_HTTP_PUBLISH_URI: "http://graylog3:9000/"
      #GRAYLOG_HTTP_EXTERNAL_URI: "http://log3.it.yudelei.com:9000/"
      GRAYLOG_ELASTICSEARCH_HOSTS: "http://opensearch1:9200,http://opensearch2:9200,http://opensearch3:9200"
      GRAYLOG_MONGODB_URI: "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/graylog"
    ports:
      - target: 9000
        published: 9000
        mode: host
      - target: 1514 # Syslog TCP
        published: 11514
        mode: host
      - target: 1514 # Syslog UDP
        published: 11514
        protocol: udp 
        mode: host
      - target: 12201 # GELF TCP
        published: 12201
        mode: host
      - target: 12201 # GELF UDP
        published: 12201
        protocol: udp
        mode: host
      - target: 5044 # Logstash
        published: 5044
        mode: host
    volumes:
      - type: bind
        source: /data-nfs/graylog/graylog3-data
        target: /usr/share/graylog/data/data
      - type: bind
        source: /data-nfs/graylog/graylog3-journal
        target: /usr/share/graylog/data/journal
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == t02-ubuntu
      restart_policy:
        condition: on-failure
        max_attempts: 3
    networks:
      - net72

networks:
  net72:
    driver: overlay

问题

当使用主机卷 Mount Bind 方式时，需要设置主机上的文件夹权限，否则会出现权限相关错误，例如：

AccessDeniedException: /usr/share/opensearch/data/performance_analyzer_enabled.conf
AccessDeniedException[/usr/share/opensearch/data/nodes]

Unable to access file /usr/share/graylog/data/journal/...

若在 System 下的 Nodes 中报如下错误，看不到其他节点状态以及信息，点击其他节点则报错，原因为未配置 GRAYLOG_HTTP_PUBLISH_URI。

System information is currently unavailable

若在 Graylog 中弹出如下警告，则表示参数设置错误，需设置一台 graylog1 的 GRAYLOG_IS_LEADER: "true"。

There was no leader Graylog server node detected in the cluster.
... is_leader = ture ...

参考

https://docs.docker.com/engine/swarm/
https://docs.docker.com/compose/intro/history/
https://go2docs.graylog.org/5-0/setting_up_graylog/server.conf.html
https://go2docs.graylog.org/current/downloading_and_installing_graylog/docker_installation.htm?Highlight=GRAYLOG_HTTP_EXTERNAL_URI
https://github.com/Graylog2/graylog2-server/issues/5700
https://go2docs.graylog.org/current/setting_up_graylog/multi-node_setup.html
https://www.mongodb.com/resources/products/compatibilities/deploying-a-mongodb-cluster-with-docker
https://www.mongodb.com/zh-cn/docs/manual/core/wiredtiger/
https://github.com/Graylog2/graylog2-server/issues/2155
https://www.mongodb.com/zh-cn/docs/manual/reference/program/mongod/
https://docs.docker.com/engine/swarm/services/
https://opensearch.org/docs/2.15/install-and-configure/install-opensearch/docker/
https://github.com/iamabrantes/Cluster-Graylog-Replicas/blob/main/stack/elastic/elastic.yml
https://github.com/Graylog2/docker-compose/blob/main/cluster/docker-compose.yml
https://gist.github.com/ruanbekker/4a9c0d250bce9f84482f2a788ce92131
https://www.mongodb.com/zh-cn/docs/manual/tutorial/troubleshoot-replica-sets/
https://medium.com/workleap/the-only-local-mongodb-replica-set-with-docker-compose-guide-youll-ever-need-2f0b74dd8384
https://medium.com/@albertorojasm95/mongodb-replicaset-with-docker-swarm-8461ecd904b0

最后修改：2024 年 09 月 27 日 11 : 34 AM
© 允许规范转载
如果觉得文章帮助了您，您可以随意赞赏。