LINUX.ORG.RU
ФорумAdmin

Не монтируется конфиг prometheus в podman

 , , , ,


0

2

Доброго времени суток! Ситуация следующая - хочу примотировать файл с конфигурацией prometheus в контейнер. Стоит Fedora 31 KDE Spin. Делаю так:

podman run -v /home/kovalev/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
На что получаю следующее:
level=info ts=2019-12-29T10:14:09.814Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-12-29T10:14:09.814Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.1, branch=HEAD, revision=8744510c6391d3ef46d8294a7e1f46e57407ab13)"
level=info ts=2019-12-29T10:14:09.814Z caller=main.go:331 build_context="(go=go1.13.5, user=root@4b1e33c71b9d, date=20191225-01:04:15)"
level=info ts=2019-12-29T10:14:09.814Z caller=main.go:332 host_details="(Linux 5.3.16-300.fc31.x86_64 #1 SMP Fri Dec 13 17:59:04 UTC 2019 x86_64 5ff79398c499 (none))"
level=info ts=2019-12-29T10:14:09.814Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)"
level=info ts=2019-12-29T10:14:09.814Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-12-29T10:14:09.816Z caller=main.go:648 msg="Starting TSDB ..."
level=info ts=2019-12-29T10:14:09.816Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-12-29T10:14:09.820Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-12-29T10:14:09.820Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:664 msg="TSDB started"
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:517 msg="Stopping scrape discovery manager..."
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:531 msg="Stopping notify discovery manager..."
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:553 msg="Stopping scrape manager..."
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:527 msg="Notify discovery manager stopped"
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:513 msg="Scrape discovery manager stopped"
level=info ts=2019-12-29T10:14:09.822Z caller=main.go:547 msg="Scrape manager stopped"
level=info ts=2019-12-29T10:14:09.822Z caller=manager.go:814 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-12-29T10:14:09.822Z caller=manager.go:820 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-12-29T10:14:09.824Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."
level=info ts=2019-12-29T10:14:09.824Z caller=main.go:718 msg="Notifier manager stopped"
level=error ts=2019-12-29T10:14:09.825Z caller=main.go:727 err="error loading config from \"/etc/prometheus/prometheus.yml\": couldn't load configuration (--config.file=\"/etc/prometheus/prometheus.yml\"): open /etc/prometheus/prometheus.yml: permission denied"
Так как делаю из под обычного юзера, и имею rootless container, предварительно поменял права на файлик командой:
podman unshare chown 65534:65534 prometheus.yml
Вот результат инспектирования контейнера:
[kovalev@tristan ~]$ podman container inspect 5ff79398c499
[
    {
        "Id": "5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed",
        "Created": "2019-12-29T22:14:09.477963528+12:00",
        "Path": "/bin/prometheus",
        "Args": [
            "--config.file=/etc/prometheus/prometheus.yml",
            "--storage.tsdb.path=/prometheus",
            "--web.console.libraries=/usr/share/prometheus/console_libraries",
            "--web.console.templates=/usr/share/prometheus/consoles"
        ],
        "State": {
            "OciVersion": "1.0.1-dev",
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2019-12-29T22:14:09.725632208+12:00",
            "FinishedAt": "2019-12-29T22:14:09.838110194+12:00",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "61bf337f29560d2c3bc5c73168014eba58eb14fdefa2e05e78a877eae29548cd",
        "ImageName": "docker.io/prom/prometheus:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/run/user/1000/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata/resolv.conf",
        "HostnamePath": "/run/user/1000/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata/hostname",
        "HostsPath": "/run/user/1000/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata/hosts",
        "StaticDir": "/home/kovalev/.local/share/containers/storage/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata",
        "OCIConfigPath": "/home/kovalev/.local/share/containers/storage/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata/config.json",
        "OCIRuntime": "crun",
        "LogPath": "/home/kovalev/.local/share/containers/storage/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata/ctr.log",
        "ConmonPidFile": "/run/user/1000/overlay-containers/5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed/userdata/conmon.pid",
        "Name": "affectionate_turing",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c38,c685",
        "ProcessLabel": "system_u:system_r:container_t:s0:c38,c685",
        "AppArmorProfile": "",
        "EffectiveCaps": null,
        "BoundingCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FSETID",
            "CAP_FOWNER",
            "CAP_MKNOD",
            "CAP_NET_RAW",
            "CAP_SETGID",
            "CAP_SETUID",
            "CAP_SETFCAP",
            "CAP_SETPCAP",
            "CAP_NET_BIND_SERVICE",
            "CAP_SYS_CHROOT",
            "CAP_KILL",
            "CAP_AUDIT_WRITE"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/home/kovalev/.local/share/containers/storage/overlay/17dc380ce4a3d9ff4c183149c9a76afa994c6274eda1626371e64bb08b0c7f6a/diff:/home/kovalev/.local/share/containers/storage/overlay/4dd9a0eb233456f41a9198f9715f27196281d12babdd05c2ef612f4c1d940eca/diff:/home/kovalev/.local/share/containers/storage/overlay/6fb46d8a6f3df49e799946d2fc451d22902409dacecb09085131b6d6ca1bdee6/diff:/home/kovalev/.local/share/containers/storage/overlay/5ee3f5c03bf4eb6f16c205e3333054a64134208d97f1115c3dbf600f67fef6b7/diff:/home/kovalev/.local/share/containers/storage/overlay/25210fff5ee214f33ff16e2d439869768f7667cfecfd16ede72fad7d93e1fb3f/diff:/home/kovalev/.local/share/containers/storage/overlay/52883c96afc7e8dd24be66ea918ea96f061f3cb891f19f541f8e7daf35557b8d/diff:/home/kovalev/.local/share/containers/storage/overlay/5e34b15110f56f964b994afe0436aa14973b61a3e85b3f3b33082dd6f1d14d55/diff:/home/kovalev/.local/share/containers/storage/overlay/286c3babfe013d1ca7cd917f7420c75d2f0e18cd90c3980b1668079133f3d63a/diff:/home/kovalev/.local/share/containers/storage/overlay/bbbe98376e3a1e598581a0ab4157ecf6d197ccae008ab72c598216f2d09d0e2f/diff:/home/kovalev/.local/share/containers/storage/overlay/3ec23c475794e68ca6290eb3cbc95b367c6dc1bdca32542c20d95ba40c0f57df/diff:/home/kovalev/.local/share/containers/storage/overlay/0f9ee83b8cfaf53335dcb96eada839e53df5f121b5f6a1a51f3f69ed7b1ea58f/diff:/home/kovalev/.local/share/containers/storage/overlay/1da8e4c8d30765bea127dc2f11a17bc723b59480f4ab5292edb00eb8eb1d96b1/diff",
                "UpperDir": "/home/kovalev/.local/share/containers/storage/overlay/8bcc287349590ec23130bb9a6113b1df46f1793302be45f659c06b446ca1c95a/diff",
                "WorkDir": "/home/kovalev/.local/share/containers/storage/overlay/8bcc287349590ec23130bb9a6113b1df46f1793302be45f659c06b446ca1c95a/work"
            }
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "402266a7e133a0891d34d52a808fefedda6785fbf9ec582f82e33181d86d437a",
                "Source": "/home/kovalev/.local/share/containers/storage/volumes/402266a7e133a0891d34d52a808fefedda6785fbf9ec582f82e33181d86d437a/_data",
                "Destination": "/prometheus",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nodev",
                    "noexec",
                    "nosuid",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Name": "",
                "Source": "/home/kovalev/prometheus.yml",
                "Destination": "/etc/prometheus/prometheus.yml",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": [],
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": ""
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/home/kovalev/.local/share/containers/storage",
            "--runroot",
            "/run/user/1000",
            "--log-level",
            "error",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/run/user/1000/libpod/tmp",
            "--runtime",
            "crun",
            "--storage-driver",
            "overlay",
            "--storage-opt",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "--events-backend",
            "journald",
            "container",
            "cleanup",
            "5ff79398c4998d3ccdf3dfaca6eb06cfc66b9b49d4b3beb8b3bb5099e3898aed"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "5ff79398c499",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "HOSTNAME=5ff79398c499",
                "container=podman",
                "HOME=/home"
            ],
            "Cmd": [
                "--config.file=/etc/prometheus/prometheus.yml",
                "--storage.tsdb.path=/prometheus",
                "--web.console.libraries=/usr/share/prometheus/console_libraries",
                "--web.console.templates=/usr/share/prometheus/consoles"
            ],
            "Image": "docker.io/prom/prometheus:latest",
            "Volumes": null,
            "WorkingDir": "/prometheus",
            "Entrypoint": "/bin/prometheus",
            "OnBuild": null,
            "Labels": {
                "maintainer": "The Prometheus Authors <prometheus-developers@googlegroups.com>"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.ContainerType": "sandbox",
                "io.kubernetes.cri-o.Created": "2019-12-29T22:14:09.477963528+12:00",
                "io.kubernetes.cri-o.TTY": "false",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15
        },
        "HostConfig": {
            "Binds": [
                "402266a7e133a0891d34d52a808fefedda6785fbf9ec582f82e33181d86d437a:/prometheus:rprivate,rw,nodev,noexec,nosuid,rbind",
                "/home/kovalev/prometheus.yml:/etc/prometheus/prometheus.yml:rw,rprivate,rbind"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [],
            "Tmpfs": {},
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "user.slice",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 4096,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1024,
                    "Hard": 1024
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        }
    }
]
Пробовал и от рута запускать, результат тот же. При этом на машине с Centos 7 и Docker все работает:
[lanadmin@galahad ~]$ docker run -v /home/lanadmin/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
level=info ts=2019-12-29T10:22:47.503Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-12-29T10:22:47.503Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.14.0, branch=HEAD, revision=edeb7a44cbf745f1d8be4ea6f215e79e651bfe19)"
level=info ts=2019-12-29T10:22:47.503Z caller=main.go:333 build_context="(go=go1.13.4, user=root@df2327081015, date=20191111-14:27:12)"
level=info ts=2019-12-29T10:22:47.504Z caller=main.go:334 host_details="(Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 11bd30da244a (none))"
level=info ts=2019-12-29T10:22:47.504Z caller=main.go:335 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-12-29T10:22:47.504Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-12-29T10:22:47.506Z caller=main.go:657 msg="Starting TSDB ..."
level=info ts=2019-12-29T10:22:47.506Z caller=web.go:496 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-12-29T10:22:47.512Z caller=head.go:535 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-12-29T10:22:47.515Z caller=head.go:583 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2019-12-29T10:22:47.520Z caller=main.go:672 fs_type=EXT4_SUPER_MAGIC
level=info ts=2019-12-29T10:22:47.520Z caller=main.go:673 msg="TSDB started"
level=info ts=2019-12-29T10:22:47.520Z caller=main.go:743 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-12-29T10:22:47.523Z caller=main.go:771 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2019-12-29T10:22:47.523Z caller=main.go:626 msg="Server is ready to receive web requests."
Вот результат docker inspect:
[lanadmin@galahad ~]$ docker container inspect 11bd30da244a
[
    {
        "Id": "11bd30da244a40490bf8a75f342c38007b10e14ae8fe564ceef0aa9fc9b0512d",
        "Created": "2019-12-29T10:22:46.94668574Z",
        "Path": "/bin/prometheus",
        "Args": [
            "--config.file=/etc/prometheus/prometheus.yml",
            "--storage.tsdb.path=/prometheus",
            "--web.console.libraries=/usr/share/prometheus/console_libraries",
            "--web.console.templates=/usr/share/prometheus/consoles"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-12-29T10:22:47.438270041Z",
            "FinishedAt": "2019-12-29T10:24:44.183894427Z"
        },
        "Image": "sha256:7317640d555e85d809d525dd7da42e703cfae67d0bcf5d6cceff699c76db9ea0",
        "ResolvConfPath": "/var/lib/docker/containers/11bd30da244a40490bf8a75f342c38007b10e14ae8fe564ceef0aa9fc9b0512d/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/11bd30da244a40490bf8a75f342c38007b10e14ae8fe564ceef0aa9fc9b0512d/hostname",
        "HostsPath": "/var/lib/docker/containers/11bd30da244a40490bf8a75f342c38007b10e14ae8fe564ceef0aa9fc9b0512d/hosts",
        "LogPath": "/var/lib/docker/containers/11bd30da244a40490bf8a75f342c38007b10e14ae8fe564ceef0aa9fc9b0512d/11bd30da244a40490bf8a75f342c38007b10e14ae8fe564ceef0aa9fc9b0512d-json.log",
        "Name": "/brave_sutherland",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/lanadmin/prometheus.yml:/etc/prometheus/prometheus.yml"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/647a453e98209b22fdeeeb32a27268e50ddc566bb96cac601c5748b1ea3f5160-init/diff:/var/lib/docker/overlay2/b8b2b15b3bd6b995e5e712e39a7cb298288c4c3d977c158d5a5d3494790b8c1d/diff:/var/lib/docker/overlay2/cdfa15ecf2577bb3c4f28d219b1f42941d3c4abeeb29d52de796258592349ec0/diff:/var/lib/docker/overlay2/0cbba385c98b1e81d0d2c67af198a2e918c23b6b707c7c4dffb68967e1097d61/diff:/var/lib/docker/overlay2/14bf36e78377f8f1ca8da154e07d3161cfe49d2e9516befc33fd2bc33d8ef6ee/diff:/var/lib/docker/overlay2/ff41f5c14922b43db5abad8dd94058f4b04c348302fabc0c60fdad5e294450d4/diff:/var/lib/docker/overlay2/591676068d5cf929289e8c88d6bc176ceea837683ac63e26492658923e577327/diff:/var/lib/docker/overlay2/c840b33aafd5368d9ba50b39ac874e771a250a384f27e1b15e4bff69f1e50caa/diff:/var/lib/docker/overlay2/399b12edbb16c741c65b157e7f81102baaf6ce36e66f00f9496d8d45f0f281e2/diff:/var/lib/docker/overlay2/287ab134ba57b42f2b3f1f69c99fc7b41de1ad409a1aed2d8aaff0fbc94ddbe5/diff:/var/lib/docker/overlay2/8f5e6ba5bac32f6faee37cbe37349781ed4933258f4bdc42ea38b297a847b2f1/diff:/var/lib/docker/overlay2/d26041fab1d13e33c458eaf19e643413b79336e8ef51d016524a5459b147c109/diff:/var/lib/docker/overlay2/4eeb9d1fd133c21617bc2b9ed8beff6ab7023791c576f17f4ce7b5c4f17b1c81/diff",
                "MergedDir": "/var/lib/docker/overlay2/647a453e98209b22fdeeeb32a27268e50ddc566bb96cac601c5748b1ea3f5160/merged",
                "UpperDir": "/var/lib/docker/overlay2/647a453e98209b22fdeeeb32a27268e50ddc566bb96cac601c5748b1ea3f5160/diff",
                "WorkDir": "/var/lib/docker/overlay2/647a453e98209b22fdeeeb32a27268e50ddc566bb96cac601c5748b1ea3f5160/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/lanadmin/prometheus.yml",
                "Destination": "/etc/prometheus/prometheus.yml",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "7b7ab1b523b0999ee3d62dd2ac4ee48b710136fcd3df61409e1f91042f13f4ca",
                "Source": "/var/lib/docker/volumes/7b7ab1b523b0999ee3d62dd2ac4ee48b710136fcd3df61409e1f91042f13f4ca/_data",
                "Destination": "/prometheus",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "11bd30da244a",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "9090/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--config.file=/etc/prometheus/prometheus.yml",
                "--storage.tsdb.path=/prometheus",
                "--web.console.libraries=/usr/share/prometheus/console_libraries",
                "--web.console.templates=/usr/share/prometheus/consoles"
            ],
            "Image": "prom/prometheus",
            "Volumes": {
                "/prometheus": {}
            },
            "WorkingDir": "/prometheus",
            "Entrypoint": [
                "/bin/prometheus"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "The Prometheus Authors <prometheus-developers@googlegroups.com>"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "f8e7ddf90331413ade34b2d1518177dc679c3dca701a153ae5dd32a83dc72d02",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/f8e7ddf90331",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "19f2a7e4d014643a0fb8d9e5ebb344de079ef67038958b0ef81864a72252ae64",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]
А теперь самое (на мой взгляд) интересное - Если смонтировать его с именем prometheus2.yml, зайти внутрь, и запустить еще один процесс прометея с новым конфигом и на другом порту - все работает.
[kovalev@tristan ~]$ podman run -d -v /home/kovalev/prometheus.yml:/etc/prometheus/prometheus2.yml prom/prometheus
8967a2a1c42b3022dfc665608b8ef083d2ae37d9a584852142fde2ce6384f149
[kovalev@tristan ~]$ podman exec -it 8967a2a1c42b3022dfc665608b8ef083d2ae37d9a584852142fde2ce6384f149 sh
/prometheus $ ls -lah /etc/prometheus/
total 16K    
drwxr-xr-x    2 nobody   nogroup     4.0K Dec 29 10:27 .
drwxr-xr-x    3 root     root        4.0K Dec 29 10:27 ..
lrwxrwxrwx    1 nobody   nogroup       39 Dec 25 03:06 console_libraries -> /usr/share/prometheus/console_libraries
lrwxrwxrwx    1 nobody   nogroup       31 Dec 25 03:06 consoles -> /usr/share/prometheus/consoles/
-rw-r--r--    1 nobody   nogroup      926 Dec 25 02:59 prometheus.yml
-rw-r--r--    1 nobody   nogroup     1.3K Dec 29 09:07 prometheus2.yml
/prometheus $ /bin/prometheus --config.file="/etc/prometheus/prometheus2.yml" --web.listen-address="0.0.0.0:9091"  
level=info ts=2019-12-29T10:30:42.922Z caller=main.go:294 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-12-29T10:30:42.922Z caller=main.go:330 msg="Starting Prometheus" version="(version=2.15.1, branch=HEAD, revision=8744510c6391d3ef46d8294a7e1f46e57407ab13)"
level=info ts=2019-12-29T10:30:42.922Z caller=main.go:331 build_context="(go=go1.13.5, user=root@4b1e33c71b9d, date=20191225-01:04:15)"
level=info ts=2019-12-29T10:30:42.922Z caller=main.go:332 host_details="(Linux 5.3.16-300.fc31.x86_64 #1 SMP Fri Dec 13 17:59:04 UTC 2019 x86_64 8967a2a1c42b (none))"
level=info ts=2019-12-29T10:30:42.922Z caller=main.go:333 fd_limits="(soft=1024, hard=1024)"
level=info ts=2019-12-29T10:30:42.922Z caller=main.go:334 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-12-29T10:30:42.924Z caller=main.go:648 msg="Starting TSDB ..."
level=info ts=2019-12-29T10:30:42.924Z caller=web.go:506 component=web msg="Start listening for connections" address=0.0.0.0:9091
level=info ts=2019-12-29T10:30:42.928Z caller=head.go:584 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-12-29T10:30:42.928Z caller=head.go:632 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2019-12-29T10:30:42.930Z caller=main.go:663 fs_type=EXT4_SUPER_MAGIC
level=info ts=2019-12-29T10:30:42.930Z caller=main.go:664 msg="TSDB started"
level=info ts=2019-12-29T10:30:42.930Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus2.yml
level=info ts=2019-12-29T10:30:42.931Z caller=main.go:762 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus2.yml
level=info ts=2019-12-29T10:30:42.931Z caller=main.go:617 msg="Server is ready to receive web requests."

Подскажите пожалуйста куда копнуть чтобы разобраться. Просто мне кажется что косяк сколько не в podman а в том что он все же несколько отличается от докера.