OS · Jorge Anaya's personal insight hub

Docker prettify docker ps output

August 1 | Docker

Everytime I run docker ps to list my containers, I find hard to understand where does the list start and where it ends:

What the! ... the port, which one? ... I almost feel like I'm seeing the matrix

I have to say, after some time your eyes get used to and begin to do weird movements but at the end you begin to understand the output, of course it shouldn't be this way, so I found there are server ways to improve the command output, let's take a look.

Use —format

Use the option --format, using this we can choose from a list of fields (see the table below) and display them in a ordered layout using the following syntax table {{.FieldName}}\t. The format option uses Go templates underneath, hey that looks familiar to me¹.

table print the field name and \t adds space between columns.

Example using --format

            

 docker ps --format 'table {{.Names}}\t
                           {{.Status}} : {{.RunningFor}}\t
                           {{.ID}}\t
                           {{.Image}}'

And we got this :
Definitely a better command output so much easy to read and understand

Taking advantage of Go Templates

We can get creative with Go Templates and lets query our containers to get network information including ports (a clean output would be to show every port in his own line but only if ports are used):

Network information

            

docker ps --format 'table {{.Names}}
                          \t{{.Status}}
                          \t{{.Networks}}
                          \n{{.ID}}
                          {{if .Ports}}
                            {{with $p := split .Ports ", "}}
                              {{range $p}}\t\t{{println .}}{{end}}
                            {{end}}
                          {{else}}
                            \t\t{{println "No Ports"}}
                          {{end}}'

Here you have, we can get all network data in a more readable way

All containers docker ps –a

Using the option ps -a will list all containers in any state. So it would be nice to add the disk space, a summary of the network use and the image from where the container derives.

All containers list

            

docker ps -a -q --format 'table {{.Names}}\t
                                {{.Status}}\t
                                {{.Size}}\n
                                {{.ID}}\t
                                {{.Image}}
                                {{if .Ports}}
                                  {{with $p := split .Ports ", "}}\t
                                    {{len $p}} port(s) on {{end}}{{- .Networks}}
                                  {{else}}\tNo Ports on {{ .Networks }}
                                {{end}}\n'

Look at this! An easier way to spot information about all containers.

Create a function to reuse the commands

Finally we can create a function to reuse these commands (if you are using WSL or Linux) inside our .bash_aliases file

Create a bash function that allows parameters

            

# Docker PS Prettify Function
function dock() {
  if [[ "$@" == "ps" ]]; then
    command docker ps --format 'table {{.Names}}\t{{.Status}} : {{.RunningFor}}\t{{.ID}}\t{{.Image}}'
  elif [[ "$@" == "psa" ]]; then
    # docker ps -a includes all containers
    command docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Size}}\n{{.ID}}\t{{.Image}}{{if .Ports}}{{with $p := split .Ports ", "}}\t{{len $p}} port(s) on {{end}}{{- .Networks}}{{else}}\tNo Ports on {{ .Networks }}{{end}}\n'
  elif [[ "$@" == "psnet" ]]; then
    # docker ps with network information
    command docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Networks}}\n{{.ID}}{{if .Ports}}{{with $p := split .Ports ", "}}{{range $p}}\t\t{{println .}}{{end}}{{end}}{{else}}\t\t{{println "No Ports"}}{{end}}'
  else
    command docker "$@"
  fi
}

To use the function just type:

dock ps: List running containers and its image name
dock psa: List all containers no matter what state and includes disk space and network information
dock psnet: List running containers with detailed network information

References

Available fields in docker

.ID: Container ID
.Image: Image ID
.Command: Quoted command
.CreatedAt: Time when the container was created.
.RunningFor: Elapsed time since the container was started.
.Ports: Exposed ports.
.Status: Container status.
.Size: Container disk size.
.Names: Container names.
.Labels: All labels assigned to the container.
.Label: Value of a specific label for this container. For example ‘{{.Label “com.docker.swarm.cpu”}}’
.Mounts: Names of the volumes mounted in this container.
.Networks: Names of the networks attached to this container.

This blog uses HUGO, and HUGO uses GoLang html and text templates. ↩︎

Docker network name is ambiguous

March 25 | Docker

The Issue

I was working on my Linux server, and I had to restart it. The docker instance was empty when the server came back, and it didn't show any container. I was in shock, and I thought I had lost my images and containers. After reviewing the system's health couldn't found a root cause, so I did a second restart 😅 and that's it the containers were back. Great! But when I tried to start my containers …

ERROR: network xxxx is ambiguous (2 matches found based on name)

The Whys

I found that docker allows repeating network names 👻; my hunch here is that when the system restarted something (maybe my container disk didn't mount properly), docker recreated the network configuration. As you can see in the following table, it duplicated every network and driver

docker network list

            

$ docker network ls
NETWORK ID NAME DRIVER SCOPE
27b6c27fdc2b bridge bridge local
33f2a0c04878 bridge bridge local
c4247d693521 host host local
9b95be563bf7 host host local
590102262bb5 none null local
25e1bf9dc86 none null local

So the solution appeared to be, just to delete one of the duplicate networks by his ID. I tried to execute the following command docker network rm [ID]

Error : bridge is a pre-defined network and cannot be removed

What the … this was more tricky than what I initially thought, at some forum, I found some arbitrary advice:

The docker service must be stopped
There shouldn't be a container using the network
And another workaround is not to have containers (delete them)

If you need to see which containers are using which network use:

Inspect network ID of a container

            

# Note that in this scenario since bridge is repeated you'll need to do it by Id:
docker network inspect [id || name]

"Internal": false,
{ "Network": "" },
"ConfigOnly": false,
"Containers": {},

So, at this point option one and two didn't work and I couldn't afford to delete all my containers which there were already configured with his own volumes and startup scripts.

Workaround and Solution

Create a new network. Use the parameter -d to specify the driver

docker network create -d bridge [new-network-name]
Disconnect the container(s) from the ambiguous network

docker network disconnect bridge [container-name]
Connect the container(s) to the new network

docker network connect [new-network-name] [container-name]
Optional. Purge our network and get rid off of the unused networks

docker network rm $(docker network ls -q)

And that's all, now we should be able to start our containers.

docker start my-happy-container

FFmpeg concat videos without reencoding

January 12 | Terminal

To concatenate some videos using ffmpeg we need to provide a text file with the word file followed by the path to the media:

VideoList.txt

            

# We can use absolute or relative paths

file '/path/to/video1.mp4'
file '/path/to/video2.mp4'
file '/path/to/video3.mp4'

Run

            

# If you use an absolute path to the file we need to add the parameter -safe 0
ffmpeg -f concat -safe 0 -i videolist.txt -c copy output.mp4

The following scripts will allow us to automate the creation of the text file

            


# Windows
(for %i in (\*.mp4) do @echo file '%i') > videolist.txt
ffmpeg -f concat -i videolist.txt -c copy output.mp4

# Bash
for f in ./\*.mp4; do echo "file '\$f'" >> videolist.txt; done
ffmpeg -f concat -i videolist.txt -c copy output.mp4

# Zsh - Using Zsh we can run the command in one single line
ffmpeg -f concat -safe 0 -i <(for f in ./\*.mp4; do echo "file '$PWD/$f'"; done) -c copy output.mp4

Sometimes would be useful to repeat the same video n times, use the following scripts

Repeat same video

            


# Windows ... in (start,step,end)
(for /l %i in (1,1,10) do @echo file './videoloop.mp4') > mylist.txt
ffmpeg -f concat -i list.txt -c copy output.mp4

# Bash
for i in {1..4}; do printf "file '%s'\n" input.mp4 >> list.txt; done
ffmpeg -f concat -i list.txt -c copy output.mp4

REFERENCES:

Terminal check if string has value

January 5 | Terminal

A basic task when you are scripting is to verify if a string is empty or not. We can do it using the following operators -n y -z

Check if empty

            

#!/bin/bash

VAR=''
if [[ -z $VAR ]]; then
echo "String is empty."
fi

Check if not empty

            

#!/bin/bash

VAR='Linuxize'
if [[ -n $VAR ]]; then
echo "String is not empty."
fi

REFERENCE: Linuxize

Clean recycle bin for all users

July 20 | Windows

The Emergency 🔔

Notification, there is an emergency, you need to connect to a remote desktop to run some transact SQL queries, you do it, press F5 and … 💥

An error occurred while executing batch. Error message is: There is not enough space on the disk....

You realize that there isn't enough space on C - nothing, zero, nada - It turns out that SSMS needs space to write some temp files on C:\Users\...\AppData\Local\Temp so it can process the results of your query.

Your first reaction, of course, is to clean temp files, your recycle bin, old documents, maybe execute the cleanup wizard in windows but if you are lucky you'll get only a few megabytes

WinDirStat

Suddenly you remember this amazing app WinDirStat, so you install it and let it do its work. Thanks to this, you'll find two issues:

Clean Recycle Bin for Everybody

Issue 1. Your partners aren't careful with their recycle bins

Open your terminal as administrator and run :

Clean recycle bin

            

rd /s c:\$Recycle.Bin

If somebody complains you could tell: Don't blame me, it was in the recycle bin for a reason

Windows Installer Folder

Issue 2. Windows is messy with the cleanup of \Windows\Installer folder

This should be addressed more carefully, it's a known issue that Windows Installer folder gets big especially when you install a lot of applications and share your machine with more users In short, the issue is that when you uninstall an app, their update / installation package is not always deleted It is supposed that you shouldn't touch this folder because could lead to update or install issues, but as always is a trade off, you should proceed carefully.

So the fix is to find orphaned installation/update files that belong to apps that are not installed anymore in the system.

Manually run the next VBA Script that shows the files that we should KEEP in the system, everything that is not in the output file could be deleted

VBA Script

            

'' Identify which patches are registered on the system, and to which
'' products those patches are installed.
''
'' Copyright (C) Microsoft Corporation. All rights reserved.
''
'' THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
'' KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
'' IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
'' PARTICULAR PURPOSE.

'Option Explicit

Dim msi : Set msi = CreateObject("WindowsInstaller.Installer")

'Output CSV header
WScript.Echo "The data format is ProductCode, PatchCode, PatchLocation"
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.CreateTextFile("output.txt", True)
objFile.WriteLine "ProductCode, PatchCode, PatchLocation"
objFile.WriteLine ""
' Enumerate all products
Dim products : Set products = msi.Products
Dim productCode

For Each productCode in products
' For each product, enumerate its applied patches
Dim patches : Set patches = msi.Patches(productCode)
Dim patchCode

    For Each patchCode in patches
    	' Get the local patch location
    	Dim location : location = msi.PatchInfo(patchCode, "LocalPackage")
        objFile.WriteLine productCode & ", " & patchCode & ", " & location

    Next

Next
WScript.Echo "Data written to output.txt, these are the registered objects and SHOULD be kept!"

Thanks to Raymond.cc for the article, you could explore the article to find other ways to clean up the windows installer folder.

Hopefully, with these two tips you could claim precious space for your computer

Map a folder to a drive in Windows

March 18 | Windows

An old DOS command will allow us to map a directory to a drive in windows. This could be useful in many scenarios (). There are two commands that we can use:

subts

            

REM to create
subst X: "C:\Users\LuckyDeveloper\MyFolderToMap"
REM to delete
subst X: /D

Drive name must be followed by a whitespace
If the target folder has whitespace in its name, use quotes “\directory withe white spaces"
Notice that at the end of the command we don't use the character “\”

✍️ This command expects that the folder is always present, be aware of this so you don't impair the performance of your machine

✍️ The map will reset with every restart, to keep it between restarts add it to windows start folder C:\Program Data\Microsoft\Windows\Start Menu\Programs\Startup

🔗 (WindowsKey + R, type: shell:common startup)

net use

net use D: \\SomeNetHost\c$\Users\LuckyDeveloper\MyFolderToMap

✍️ net use is more fault-tolerant about folder availability, so is better to use with network shared resources.

OS

TOPICS

August 2020

Use —format

Taking advantage of Go Templates

All containers docker ps –a

Create a function to reuse the commands

March 2020

The Issue

The Whys

Workaround and Solution

January 2020

July 2019

The Emergency 🔔

WinDirStat

Clean Recycle Bin for Everybody

Windows Installer Folder

March 2019

subts

net use