Du and Find on Following Dirs Took Will Not Log Again for This Container Unless Duration Exceed

Almost storage drivers

Estimated reading time: twenty minutes

To utilise storage drivers finer, it's important to know how Docker builds and stores images, and how these images are used by containers. You can use this information to make informed choices about the best way to persist data from your applications and avoid performance problems along the manner.

Storage drivers versus Docker volumes

Docker uses storage drivers to store image layers, and to store information in the writable layer of a container. The container's writable layer does not persist afterwards the container is deleted, but is suitable for storing ephemeral data that is generated at runtime. Storage drivers are optimized for infinite efficiency, just (depending on the storage driver) write speeds are lower than native file system performance, especially for storage drivers that apply a re-create-on-write filesystem. Write-intensive applications, such as database storage, are impacted by a performance overhead, specially if pre-existing data exists in the read-only layer.

Use Docker volumes for write-intensive data, data that must persist beyond the container'south lifespan, and data that must exist shared betwixt containers. Refer to the volumes department to learn how to use volumes to persist data and improve operation.

Images and layers

A Docker image is built up from a series of layers. Each layer represents an instruction in the paradigm'south Dockerfile. Each layer except the very final ane is read-just. Consider the following Dockerfile:

                          # syntax=docker/dockerfile:one              FROM                              ubuntu:18.04              Characterization                              org.opencontainers.paradigm.authors="org@example.com"              Copy                              . /app              RUN              brand /app              RUN              rm              -r              $Domicile/.enshroud              CMD                              python /app/app.py                      

This Dockerfile contains four commands. Commands that modify the filesystem create a layer. TheFROM statement starts out past creating a layer from the ubuntu:18.04 prototype. The Label command just modifies the image's metadata, and does not produce a new layer. The COPY command adds some files from your Docker client's current directory. The showtime RUN command builds your application using the make command, and writes the result to a new layer. The 2d RUN command removes a cache directory, and writes the result to a new layer. Finally, the CMD education specifies what command to run within the container, which only modifies the image's metadata, which does not produce an image layer.

Each layer is only a set of differences from the layer before information technology. Note that both adding, and removing files volition event in a new layer. In the case above, the $Dwelling house/.enshroud directory is removed, but volition still be available in the previous layer and add up to the image's full size. Refer to the All-time practices for writing Dockerfiles and apply multi-stage builds sections to learn how to optimize your Dockerfiles for efficient images.

The layers are stacked on top of each other. When you create a new container, you add together a new writable layer on top of the underlying layers. This layer is often chosen the "container layer". All changes made to the running container, such equally writing new files, modifying existing files, and deleting files, are written to this sparse writable container layer. The diagram below shows a container based on an ubuntu:15.04 paradigm.

A storage commuter handles the details about the way these layers interact with each other. Different storage drivers are bachelor, which take advantages and disadvantages in unlike situations.

Container and layers

The major difference betwixt a container and an paradigm is the top writable layer. All writes to the container that add new or change existing data are stored in this writable layer. When the container is deleted, the writable layer is also deleted. The underlying prototype remains unchanged.

Because each container has its ain writable container layer, and all changes are stored in this container layer, multiple containers tin can share access to the aforementioned underlying image and all the same have their own data state. The diagram below shows multiple containers sharing the same Ubuntu xv.04 epitome.

Docker uses storage drivers to manage the contents of the paradigm layers and the writable container layer. Each storage driver handles the implementation differently, simply all drivers utilise stackable epitome layers and the copy-on-write (Cow) strategy.

Note

Utilise Docker volumes if you need multiple containers to have shared admission to the verbal same data. Refer to the volumes section to learn almost volumes.

Container size on deejay

To view the approximate size of a running container, yous can use the docker ps -s control. Two different columns relate to size.

• size: the corporeality of data (on disk) that is used for the writable layer of each container.
• virtual size: the corporeality of data used for the read-only paradigm data used by the container plus the container'southward writable layer size. Multiple containers may share some or all read-but image data. 2 containers started from the same image share 100% of the read-only data, while two containers with different images which have layers in common share those mutual layers. Therefore, you can't just full the virtual sizes. This over-estimates the total disk usage by a potentially not-trivial corporeality.

The full disk infinite used by all of the running containers on disk is some combination of each container's size and the virtual size values. If multiple containers started from the same exact image, the total size on deejay for these containers would be SUM (size of containers) plus one paradigm size (virtual size- size).

This also does not count the following boosted ways a container tin can have upwards disk space:

• Deejay space used for log files stored past the logging-driver. This can be not-little if your container generates a big amount of logging data and log rotation is not configured.
• Volumes and demark mounts used by the container.
• Disk infinite used for the container's configuration files, which are typically small.
• Memory written to disk (if swapping is enabled).
• Checkpoints, if you're using the experimental checkpoint/restore feature.

The re-create-on-write (Moo-cow) strategy

Copy-on-write is a strategy of sharing and copying files for maximum efficiency. If a file or directory exists in a lower layer within the image, and another layer (including the writable layer) needs read access to information technology, it just uses the existing file. The get-go time another layer needs to change the file (when building the epitome or running the container), the file is copied into that layer and modified. This minimizes I/O and the size of each of the subsequent layers. These advantages are explained in more depth below.

Sharing promotes smaller images

When you lot utilize docker pull to pull downwardly an prototype from a repository, or when yous create a container from an image that does non nevertheless exist locally, each layer is pulled downwards separately, and stored in Docker'southward local storage area, which is unremarkably /var/lib/docker/ on Linux hosts. You can see these layers being pulled in this instance:

                          $              docker pull ubuntu:18.04              18.04: Pulling from library/ubuntu f476d66f5408: Pull complete 8882c27f669e: Pull complete d9af21273955: Pull consummate f5029279ec12: Pull complete Digest: sha256:ab6cb8de3ad7bb33e2534677f865008535427390b117d7939193f8d1a6613e34 Condition: Downloaded newer image for ubuntu:18.04                                    

Each of these layers is stored in its own directory inside the Docker host's local storage area. To examine the layers on the filesystem, list the contents of /var/lib/docker/<storage-driver>. This example uses the overlay2 storage driver:

                          $              ls              /var/lib/docker/overlay2              16802227a96c24dcbeab5b37821e2b67a9f921749cd9a2e386d5a6d5bc6fc6d3 377d73dbb466e0bc7c9ee23166771b35ebdbe02ef17753d79fd3571d4ce659d7 3f02d96212b03e3383160d31d7c6aeca750d2d8a1879965b89fe8146594c453d ec1ec45792908e90484f7e629330666e7eee599f08729c93890a7205a6ba35f5 l                                    

The directory names practise not correspond to the layer IDs.

Now imagine that you have two dissimilar Dockerfiles. Y'all use the first i to create an image called top/my-base of operations-image:1.0.

                          # syntax=docker/dockerfile:1              FROM                              alpine              RUN              apk add together              --no-enshroud              bash                      

The second one is based on elevation/my-base-prototype:one.0, simply has some additional layers:

                          # syntax=docker/dockerfile:1              FROM                              acme/my-base-image:1.0              Copy                              . /app              RUN              chmod +x /app/hello.sh              CMD                              /app/howdy.sh                      

The second paradigm contains all the layers from the starting time epitome, plus new layers created by the Copy and RUN instructions, and a read-write container layer. Docker already has all the layers from the starting time image, so information technology does not need to pull them again. The two images share any layers they have in common.

If you build images from the ii Dockerfiles, you can use docker paradigm ls and docker epitome history commands to verify that the cryptographic IDs of the shared layers are the same.

1. Brand a new directory cow-examination/ and change into it.

2. Within cow-test/, create a new file called how-do-you-do.sh with the following contents:

                                     #!/usr/bin/env bash                  echo                  "Hello world"                              

3. Copy the contents of the beginning Dockerfile above into a new file called Dockerfile.base.

4. Copy the contents of the second Dockerfile to a higher place into a new file chosen Dockerfile.

5. Within the cow-examination/ directory, build the offset image. Don't forget to include the final . in the control. That sets the PATH, which tells Docker where to expect for any files that need to be added to the image.

                                     $                  docker build                  -t                  acme/my-base-image:1.0                  -f                  Dockerfile.base                  .                  [+] Building half dozen.0s (11/xi) FINISHED                                    =>                  [internal] load build definition from Dockerfile.base                                      0.4s                  =>                  =>                  transferring dockerfile: 116B                                                           0.0s                  =>                  [internal] load .dockerignore                                                              0.3s                  =>                  =>                  transferring context: 2B                                                                0.0s                  =>                  resolve image config                  for                  docker.io/docker/dockerfile:i                                     1.5s                  =>                  [auth] docker/dockerfile:pull token                  for                  registry-ane.docker.io                               0.0s                  =>                  Buried docker-image://docker.io/docker/dockerfile:1@sha256:9e2c9eca7367393aecc68795c671... 0.0s                  =>                  [internal] load .dockerignore                                                              0.0s                  =>                  [internal] load build definition from Dockerfile.base                                      0.0s                  =>                  [internal] load metadata                  for                  docker.io/library/alpine:latest                               0.0s                  =>                  Buried                  [i/2] FROM docker.io/library/alpine                                                 0.0s                  =>                  [2/ii] RUN apk add together                  --no-cache                  fustigate                                                          3.1s                  =>                  exporting to paradigm                                                                         0.2s                  =>                  =>                  exporting layers                                                                        0.2s                  =>                  =>                  writing image sha256:da3cf8df55ee9777ddcd5afc40fffc3ead816bda99430bad2257de4459625eaa   0.0s                  =>                  =>                  naming to docker.io/acme/my-base of operations-epitome:one.0                                              0.0s                              

6. Build the 2d prototype.

                                     $                  docker build                  -t                  elevation/my-terminal-image:1.0                  -f                  Dockerfile                  .                                      [+] Building 3.6s (12/12) FINISHED                                    =>                  [internal] load build definition from Dockerfile                                            0.1s                  =>                  =>                  transferring dockerfile: 156B                                                            0.0s                  =>                  [internal] load .dockerignore                                                               0.1s                  =>                  =>                  transferring context: 2B                                                                 0.0s                  =>                  resolve image config                  for                  docker.io/docker/dockerfile:ane                                      0.5s                  =>                  Cached docker-paradigm://docker.io/docker/dockerfile:1@sha256:9e2c9eca7367393aecc68795c671...  0.0s                  =>                  [internal] load .dockerignore                                                               0.0s                  =>                  [internal] load build definition from Dockerfile                                            0.0s                  =>                  [internal] load metadata                  for                  docker.io/acme/my-base of operations-image:1.0                               0.0s                  =>                  [internal] load build context                                                               0.2s                  =>                  =>                  transferring context: 340B                                                               0.0s                  =>                  [1/3] FROM docker.io/pinnacle/my-base-image:ane.0                                                 0.2s                  =>                  [2/3] Re-create                  .                  /app                                                                           0.1s                  =>                  [3/iii] RUN chmod +x /app/hello.sh                                                            0.4s                  =>                  exporting to image                                                                          0.1s                  =>                  =>                  exporting layers                                                                         0.1s                  =>                  =>                  writing image sha256:8bd85c42fa7ff6b33902ada7dcefaaae112bf5673873a089d73583b0074313dd    0.0s                  =>                  =>                  naming to docker.io/superlative/my-final-image:ane.0                                              0.0s                              

7. Check out the sizes of the images:

                                     $                  docker prototype                  ls                                      REPOSITORY             TAG     IMAGE ID         CREATED               SIZE height/my-terminal-paradigm    1.0     8bd85c42fa7f     About a minute ago    vii.75MB pinnacle/my-base-image     i.0     da3cf8df55ee     2 minutes ago         7.75MB                                                

8. Check out the history of each image:

                                     $                  docker image                  history                  summit/my-base-image:1.0                                      Image          CREATED         CREATED BY                                      SIZE      COMMENT                                    da3cf8df55ee   5 minutes ago   RUN /bin/sh -c apk add --no-cache bash #                  bui…   ii.15MB    buildkit.dockerfile.v0                  <missing>                  7 weeks ago     /bin/sh                  -c                  #(nop)  CMD ["/bin/sh"]              0B                  <missing>                  7 weeks ago     /bin/sh                  -c                  #(nop) ADD file:f278386b0cef68136…   5.6MB                              

   Some steps do non have a size (0B), and are metadata-only changes, which practice non produce an image layer and do not accept up any size, other than the metadata itself. The output above shows that this image consists of 2 prototype layers.

                                     $                  docker image                  history                  acme/my-final-epitome:one.0                                      IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT 8bd85c42fa7f   3 minutes ago   CMD ["/bin/sh" "-c" "/app/hello.sh"]            0B        buildkit.dockerfile.v0                                    <missing>                  3 minutes ago   RUN /bin/sh                  -c                  chmod +x /app/hello.sh                  # buil…   39B       buildkit.dockerfile.v0                  <missing>                  iii minutes ago   Copy                  .                  /app                  # buildkit                          222B      buildkit.dockerfile.v0                  <missing>                  4 minutes ago   RUN /bin/sh                  -c                  apk add                  --no-cache                  bash                  # bui…   two.15MB    buildkit.dockerfile.v0                  <missing>                  vii weeks ago     /bin/sh                  -c                  #(nop)  CMD ["/bin/sh"]              0B                  <missing>                  7 weeks ago     /bin/sh                  -c                  #(nop) Add file:f278386b0cef68136…   v.6MB                              

   Notice that all steps of the showtime image are likewise included in the last image. The final image includes the two layers from the offset prototype, and two layers that were added in the second paradigm.

   What are the <missing> steps?

   The <missing> lines in the docker history output bespeak that those steps were either built on another organization and part of the alpine image that was pulled from Docker Hub, or were congenital with BuildKit as builder. Before BuildKit, the "archetype" builder would produce a new "intermediate" image for each step for caching purposes, and the IMAGE column would evidence the ID of that image. BuildKit uses its own caching mechanism, and no longer requires intermediate images for caching. Refer to build images with BuildKit to learn more about other enhancements made in BuildKit.

9. Bank check out the layers for each paradigm

   Use the docker paradigm audit control to view the cryptographic IDs of the layers in each epitome:

                                     $                  docker image inspect                  --format                  "{{json .RootFS.Layers}}"                  elevation/my-base-image:one.0                  [   "sha256:72e830a4dff5f0d5225cdc0a320e85ab1ce06ea5673acfe8d83a7645cbd0e9cf",   "sha256:07b4a9068b6af337e8b8f1f1dae3dd14185b2c0003a9a1f0a6fd2587495b204a" ]                                                

                                     $                  docker image audit                  --format                  "{{json .RootFS.Layers}}"                  acme/my-final-image:1.0                  [   "sha256:72e830a4dff5f0d5225cdc0a320e85ab1ce06ea5673acfe8d83a7645cbd0e9cf",   "sha256:07b4a9068b6af337e8b8f1f1dae3dd14185b2c0003a9a1f0a6fd2587495b204a",   "sha256:cc644054967e516db4689b5282ee98e4bc4b11ea2255c9630309f559ab96562e",   "sha256:e84fb818852626e89a09f5143dbc31fe7f0e0a6a24cd8d2eb68062b904337af4" ]                                                

   Notice that the first two layers are identical in both images. The second image adds ii additional layers. Shared image layers are but stored once in /var/lib/docker/ and are besides shared when pushing and pulling and prototype to an epitome registry. Shared epitome layers can therefore reduce network bandwidth and storage.

   Tip: format output of Docker commands with the --format option

   The examples above use the docker image inspect command with the --format option to view the layer IDs, formatted as a JSON assortment. The --format option on Docker commands can exist a powerful feature that allows you to extract and format specific data from the output, without requiring additional tools such as awk or sed. To learn more than about formatting the output of docker commands using the --format flag, refer to the format control and log output section. We besides pretty-printed the JSON output using the jq utility for readability.

Copying makes containers efficient

When you showtime a container, a sparse writable container layer is added on pinnacle of the other layers. Any changes the container makes to the filesystem are stored hither. Any files the container does not change do non get copied to this writable layer. This means that the writable layer is every bit small as possible.

When an existing file in a container is modified, the storage driver performs a copy-on-write functioning. The specifics steps involved depend on the specific storage driver. For the overlay2, overlay, and aufs drivers, the copy-on-write operation follows this rough sequence:

• Search through the paradigm layers for the file to update. The process starts at the newest layer and works downwardly to the base layer one layer at a time. When results are found, they are added to a enshroud to speed future operations.
• Perform a copy_up operation on the outset copy of the file that is found, to copy the file to the container'due south writable layer.
• Any modifications are made to this copy of the file, and the container cannot see the read-just copy of the file that exists in the lower layer.

Btrfs, ZFS, and other drivers handle the copy-on-write differently. You lot can read more nigh the methods of these drivers subsequently in their detailed descriptions.

Containers that write a lot of data consume more infinite than containers that do non. This is considering most write operations swallow new space in the container'due south sparse writable top layer. Note that changing the metadata of files, for example, changing file permissions or ownership of a file, can also result in a copy_up operation, therefore duplicating the file to the writable layer.

Tip: Use volumes for write-heavy applications

For write-heavy applications, yous should not store the data in the container. Applications, such as write-intensive database storage, are known to exist problematic particularly when pre-existing data exists in the read-only layer.

Instead, use Docker volumes, which are independent of the running container, and designed to be efficient for I/O. In addition, volumes can be shared amongst containers and exercise not increment the size of your container's writable layer. Refer to the utilise volumes section to learn about volumes.

A copy_up operation can incur a noticeable performance overhead. This overhead is different depending on which storage driver is in use. Big files, lots of layers, and deep directory trees tin can make the touch on more noticeable. This is mitigated by the fact that each copy_up functioning but occurs the first time a given file is modified.

To verify the way that copy-on-write works, the following procedures spins up 5 containers based on the pinnacle/my-concluding-image:1.0 epitome we built earlier and examines how much room they take up.

1. From a last on your Docker host, run the following docker run commands. The strings at the end are the IDs of each container.

                                     $                  docker run                  -dit                  --proper name                  my_container_1 summit/my-final-image:1.0 fustigate                  \                  &&                  docker run                  -dit                  --proper name                  my_container_2 acme/my-terminal-image:1.0 fustigate                  \                  &&                  docker run                  -dit                  --proper noun                  my_container_3 acme/my-final-prototype:1.0 bash                  \                  &&                  docker run                  -dit                  --name                  my_container_4 acme/my-last-image:1.0 fustigate                  \                  &&                  docker run                  -dit                  --name                  my_container_5 acme/my-final-image:1.0 fustigate                                      40ebdd7634162eb42bdb1ba76a395095527e9c0aa40348e6c325bd0aa289423c a5ff32e2b551168b9498870faf16c9cd0af820edf8a5c157f7b80da59d01a107 3ed3c1a10430e09f253704116965b01ca920202d52f3bf381fbb833b8ae356bc 939b3bf9e7ece24bcffec57d974c939da2bdcc6a5077b5459c897c1e2fa37a39 cddae31c314fbab3f7eabeb9b26733838187abc9a2ed53f97bd5b04cd7984a5a                                                

2. Run the docker ps command with the --size pick to verify the five containers are running, and to see each container's size.

                                     $                  docker ps                  --size                  --format                  "table {{.ID}}                  \t                  {{.Image}}                  \t                  {{.Names}}                  \t                  {{.Size}}"                                      CONTAINER ID   IMAGE                     NAMES            SIZE cddae31c314f   acme/my-final-image:1.0   my_container_5   0B (virtual 7.75MB) 939b3bf9e7ec   acme/my-final-image:1.0   my_container_4   0B (virtual 7.75MB) 3ed3c1a10430   acme/my-final-image:1.0   my_container_3   0B (virtual vii.75MB) a5ff32e2b551   acme/my-terminal-epitome:1.0   my_container_2   0B (virtual 7.75MB) 40ebdd763416   acme/my-final-image:1.0   my_container_1   0B (virtual vii.75MB)                                                

   The output above shows that all containers share the epitome's read-only layers (7.75MB), merely no information was written to the container's filesystem, so no boosted storage is used for the containers.

   Advanced: metadata and logs storage used for containers

   Note: This step requires a Linux automobile, and does not work on Docker Desktop for Mac or Docker Desktop for Windows, equally it requires access to the Docker Daemon'south file storage.

   While the output of docker ps provides y'all information most disk space consumed by a container'southward writable layer, it does not include information about metadata and log-files stored for each container.

   More details can be obtained by exploring the Docker Daemon's storage location (/var/lib/docker by default).

                                         $                    sudo                    du                    -sh                    /var/lib/docker/containers/*                                          36K  /var/lib/docker/containers/3ed3c1a10430e09f253704116965b01ca920202d52f3bf381fbb833b8ae356bc 36K  /var/lib/docker/containers/40ebdd7634162eb42bdb1ba76a395095527e9c0aa40348e6c325bd0aa289423c 36K  /var/lib/docker/containers/939b3bf9e7ece24bcffec57d974c939da2bdcc6a5077b5459c897c1e2fa37a39 36K  /var/lib/docker/containers/a5ff32e2b551168b9498870faf16c9cd0af820edf8a5c157f7b80da59d01a107 36K  /var/lib/docker/containers/cddae31c314fbab3f7eabeb9b26733838187abc9a2ed53f97bd5b04cd7984a5a                                                      

   Each of these containers simply takes up 36k of infinite on the filesystem.

3. Per-container storage

   To demonstrate this, run the following control to write the word 'hi' to a file on the container'southward writable layer in containers my_container_1, my_container_2, and my_container_3:

                                                         $                  for                  i                  in                  {1..3}                  ;                  do                  docker                  exec                  my_container_$i                  sh                  -c                  'printf hello > /out.txt'                  ;                  washed                              

   Running the docker ps command once again afterward shows that those containers now swallow 5 bytes each. This data is unique to each container, and non shared. The read-only layers of the containers are not affected, and are nevertheless shared by all containers.

                                                         $                  docker ps                  --size                  --format                  "tabular array {{.ID}}                  \t                  {{.Image}}                  \t                  {{.Names}}                  \t                  {{.Size}}"                                      CONTAINER ID   Prototype                     NAMES            SIZE  cddae31c314f   pinnacle/my-final-image:1.0   my_container_5   0B (virtual 7.75MB)  939b3bf9e7ec   acme/my-final-image:1.0   my_container_4   0B (virtual 7.75MB)  3ed3c1a10430   acme/my-final-image:ane.0   my_container_3   5B (virtual 7.75MB)  a5ff32e2b551   superlative/my-final-image:one.0   my_container_2   5B (virtual 7.75MB)  40ebdd763416   acme/my-final-epitome:1.0   my_container_1   5B (virtual vii.75MB)                                                

The examples in a higher place illustrate how copy-on-write filesystems assistance making containers efficient. Non only does copy-on-write save infinite, but it also reduces container start-upward fourth dimension. When you create a container (or multiple containers from the aforementioned image), Docker only needs to create the thin writable container layer.

If Docker had to brand an unabridged copy of the underlying image stack each time it created a new container, container create times and disk space used would be significantly increased. This would be similar to the style that virtual machines work, with one or more virtual disks per virtual machine. The vfs storage does not provide a Cow filesystem or other optimizations. When using this storage driver, a total copy of the epitome's data is created for each container.

• Volumes
• Select a storage commuter

container, storage, driver, AUFS, btrfs, devicemapper, overlayfs, vfs, zfs

inoueansuchan.blogspot.com

Source: https://docs.docker.com/storage/storagedriver/

Bagikan Artikel ini