Finland

Blue and White, what could go wrong?

Most people wouldn’t argue with this statement: The flag of Finland, displayed above, has two colors (blue and white). Let’s see if a computer would agree with that. We’ll carry out the same analysis we did above. This time we’ll create our own function for reusability.

color_count <- function(path) {
  # load image and select the original rgb representation
  image_list <- colordistance::loadImage(path)
  image_rgb <- image_list[["original.rgb"]]
  
  hex_values <- apply(image_rgb, MARGIN = c(1,2), function(x) rgb(x[1], x[2], x[3])) 

  tabled_hex_values <- table(c(hex_values))

  # return in dataframe format
  tabled_hex_values |> 
    as_tibble(.name_repair = make.names) |> 
    arrange(desc(n)) |> 
    rename(hex_value = X, cnt = n) |> 
    mutate(percent_of_total = cnt/sum(cnt))
}

color_count("images/flag_images/fi.png")
## Warning in removeBackground(img, lower = lower, upper = upper, quietly =
## alpha.message, : No transparent pixels in image
## # A tibble: 38 × 3
##    hex_value    cnt percent_of_total
##    <chr>      <int>            <dbl>
##  1 #FFFFFF   149464          0.597  
##  2 #003580    94228          0.377  
##  3 #000077     1344          0.00537
##  4 #173B82     1060          0.00424
##  5 #6E7BA2     1056          0.00422
##  6 #FEFFFF     1056          0.00422
##  7 #00006F      284          0.00113
##  8 #E0E2E8      282          0.00113
##  9 #FDFDFD      282          0.00113
## 10 #1B3C82      280          0.00112
## # ℹ 28 more rows

The computer is a lot closer to agreeing with us than with the case of the brown eye. It finds that 60% of the image is white, and about 39% is a particular shade of blue. However, what the hell?! It somehow identified 5 other colors, none of which account for more than 1% of the total image. To see what’s going on here, let’s zoom in on the part of the flag where the blue crosses.

Oh, c’mon!

When you zoom in there really close you can see that the colors blend at the edges to form a few different shades of blue. Why this is the case, I’m not sure. If I had to guess, it would be related to the PNG file format, which will pixelate when resized or converted from a vector based graphic. In any case, it certainly complicates something that I thought would be relatively straight forward. Such is life, right? Let’s trudge on.

In the case of Finland, we can easily use our judgement and say the the first two colors, which comprise over 99% of the image, are the only two colors that matter. Unfortunately, this simple rule won’t work for our entire set of flags. If we rule out any hex value comprising less than 1% of the total image, we will get false negatives and false positives. A false negative can be seen on the flag of Portugal. Most people would say this flag is red, green, yellow, white and blue. However, the blue shields on the flag turn out to represent only 0.4% of the total, so our simple 1% threshold would fail to detect a color that human intuition easily detects.

Portugal. The Flag

An example of a false positive would be Swaziland. I’d say the flag is blue, yellow, red, black, and white.

One of the sweetest flags in the world

But when we use the 1% cutoff, we see two different shades of blue. They’re almost imperceptible to the human eye, and one (#3E5EB9) represents 36% whereas the other (#3D5DA7) only represents 1.8%.

swaziland_path <- file.path("images/flag_images/sz.png")
color_count(swaziland_path)
## Warning in removeBackground(img, lower = lower, upper = upper, quietly =
## alpha.message, : No transparent pixels in image
## # A tibble: 4,080 × 3
##    hex_value   cnt percent_of_total
##    <chr>     <int>            <dbl>
##  1 #3E5EB9   92160          0.338  
##  2 #B10C0C   78230          0.287  
##  3 #FFD900   30361          0.111  
##  4 #FFFFFF   14466          0.0531 
##  5 #000000   14053          0.0515 
##  6 #3D5DA7    2250          0.00825
##  7 #FED800    1290          0.00473
##  8 #FFDB00    1287          0.00472
##  9 #AB000D    1280          0.00469
## 10 #FFDF00     941          0.00345
## # ℹ 4,070 more rows

Clearly, these should both be treated as “blue”, right? Let’s look into how we might fix that.

Illuminate me, please

I’m sure we’ve all at some point seen some version of the color spectrum. If not, below lies a famous one which was created long before modern computers were around, in 1931 by the International Commission on Illumination. Yes, I told you we’d get to that and yes, it is a real thing. In 1931, this wacky bunch convened to decide on the world’s first quantitative measure of color. What I would give to go back in time. . . Anyway, the colorspace they settled on was based on the three types of cones in our eyes which are receptive to red, green, and blue. Therefore, using the scientific knowledge of the time, they defined all colors as combinations of some amount of red, green, and/or blue. I’m trying to stay out of the weeds here, and to be honest this stuff is a little over my head, but I’ll include some interesting articles at the bottom of this essay for readers who want to get in the weeds.

1931 CIE Color Space

Having a quantitative scale for colors yields two cool results. One is that we can better communicate colors. So a graphic designer doesn’t have to call the printer and say “can you reprint and add just a little more blue?”. She can just relay the RGB values she wants and expect them to be rendered faithfully. Secondly, where there’s numbers there’s math. With the RGB scale, we can perform all our favorite math things with colors like addition, subtraction, distance, multiplication, clustering, binning, etc.

Let’s go back to our Swaziland example, where I mentioned we have two very similar shades of blue, leading to a false positive when we use our naive rule of filtering out colors at the 1% level. Our brains told us we were looking at just one blue, and with the RGB colorspace we can now speak objectively about them. Below is a glimpse of a random sample of the pixels in the Swaziland flag, plotted in the three dimensional RGB space. There’s a simple function in the colordistance library to do this called plotPixels.

loadImage(swaziland_path) |> 
  plotPixels(n = 1000, lower = NULL, upper = NULL)
## Warning in removeBackground(img, lower = lower, upper = upper, quietly =
## alpha.message, : No transparent pixels in image

We can see the blue values, and we can also see that they’re very close in the RGB 3 dimensional space. Don’t you just want to reach your hand in there and squeeze them together? The same is noticeable for black, brown, grey, and red colors. Normally, when we have continuous data like this and we want to reduce it into something more discrete, we have two go-to methods: Clustering and Binning. And of course, the colordistance package has functions for clustering and binning colors (what a great library!). Which one do we want? Well think about it. Clustering colors, which is done using k-means, requires us to state the number of clusters a priori. But our whole goal of this essay is to have the computer tell us how many colors there are. So we want to stay away from any method where we tell the computer to work with a number we’re giving it ahead of time. Binning on the other hand, is a technique that divides a space into equal sized ranges, and counts the number of observations in each range. To see what I mean, let’s bin the colors of the Swaziland flag.

sz <- loadImage(swaziland_path) |> 
  getImageHist(bins = c(3,3,2), lower = NULL, upper = NULL, title = "Swaziland Flag Color Bins")
## RGB and HSV are device-dependent, perceptually non-uniform color spaces. See 'Color spaces' vignette for more information.
## Warning in removeBackground(img, lower = lower, upper = upper, quietly =
## alpha.message, : No transparent pixels in image
## 
## Using 3*3*2 = 18 bins

The function getImageHist divides the RGB space into as many as 27 mini-cubes (although I found that using 18 works the best with the flag data). Any pixels in a given cube get averaged together, counted, and displayed on a bar chart. And as you can see, this method solved our false positive, merging the two very close blue pixels into one blue color. So this color binning method seems like a great tool to get us closer to answering the question at hand: Which flag has the most colors?

Of course, color binning is not a panacea. In the case of Swaziland, note that there are still some very small bins representing colors we probably would not subjectively say exist as distinct colors in the Swaziland flag. One example is the dark red bin representing the second bar. This color is probably a result of the pixelation around the red/blue borders of the flag. It’s not something we’re supposed to see distinctly. So if we want to use this binning method to count colors, we’ll still have to find some cutoff value to throw away these edge colors.

There’s another thing I haven’t brought up yet, too. I know, you’re thinking this is getting too long, but hopefully some part of you finds this entertaining too. Remember, the journey is the destination.

Beware of Seals: A digression

Photo from http://arresteddevelopment.wikia.com/wiki/Seal

No, I’m not talking about seals that may have developed a taste for mammal blood. I’m talking about something far scarier, murkier, and problematic. You see, while most flags are just a nice and simple arrangement of colorful shapes, some countries decided to buck that trend and use illustrative seals on their flags. Why is this a problem for us? Let me show you. Exhibit A: Nicaragua

While some might say this flag is blue and white, it’s also got this colorful seal smack in the middle with a RAINBOW of all things. So what do we make of this? Where do we draw the line? This isn’t a one-off either, a lot of flags have seals and they are pretty damn colorful. Some, like Mexico, are quite illustrative with color gradients that even our old friend Betsy Ross would have told us can’t be sewn. The Mexican flag brings up a lot of other questions, too. How much of the color and design of the eagle is up to the artist, and how much is government mandated. At what size/scale does the flag need to achieve what level of detail? Are there government guidelines outlining this?

The finest seamstress in the land can’t achieve these details

It just keeps getting messier and messier, but we’re already in too deep so don’t look back. Let’s get back to the problem we noticed before this digression, which is that our color binning technique still leaves behind trace levels of colors that a human would never notice. You can probably guess that flags with seals will amplify this problem, since seals have a lot of detail which leads to gradation and blurred lines. Let’s call these the “long tail of colors”.

Cutting the tail

Towards the beginning of this essay I proposed a dumb heuristic to deal with the long tail of colors. I hoped that removing any color comprising less than 1% of the total image might work, but in practice it led to some false positives and negatives (Portugal and Swaziland being the two examples we used). Let’s see if we can use the color binned data to come up with a more principled threshold. First though, I realize we haven’t touched upon something extremely important: our data set.

The flag data I’ve been using comes from flagpedia. You’ll see that they have an option to download all the flags of the world in one zip file, and that they have 4 different image resolutions. I used the high resolution set for this analysis. It’s large enough to show good detail on the flags that have seals, while not too big to overwhelm your CPU on tasks like pixel counting and binning colors (if you have a GPU I’m sure you could use that, but if you have a GPU you probably stopped reading this awhile back and went back to your deep learning project). The files in the zip are PNGs and each country is identified by it’s two letter ISO code.

The first thing we want to do is prepare a nice data frame with all this data in it. I simply created a vector of flag images by reading the files in the directory and removing the .png file ending. Then, using the countrycode package, I was able to get the official country names for all the ISO codes. I also added the continent for good measure.

flag_dir <- file.path("images/flag_images/")
flag_images <- list.files(flag_dir)

library(countrycode)
flag_df <- tibble(
  flag_file_name = flag_images,
  iso_code = stringr::str_remove(flag_images, ".png"),
  country = countrycode(iso_code, origin = "iso2c", destination = "country.name"),
  continent = countrycode(iso_code, origin = "iso2c", destination = "continent")
)

Next, we want to apply the color binning sequence to all our flags. This takes a little while with the large images, so if you’re playing along at home you may want to use smaller images or just wait the ten minutes it takes with the large ones. I decided to write a function that takes the filename as an argument, loads it, bins and counts the pixels, converts the RGB values into its hex value, and calculates the percentage of the total each color bin comprises.

Then I simply map that function to each file in our data frame.

pixel_bins <- function(iso_code, bin_vec = c(3,3,2)) {
  
  flag_png = str_c(iso_code, ".png")

  color_bins <- colordistance::getImageHist(file.path(flag_dir, flag_png), 
                                            bins = bin_vec, 
                                            plotting = F, 
                                            lower = NULL, 
                                            upper = NULL
                                            ) |> 
    as_tibble() |> 
    mutate(hex = rgb(r, g, b)) |> 
    arrange(desc(Pct))
  
  return(color_bins)
  
}

flag_color_df <- flag_df |> 
  mutate(color_bins = purrr::map(iso_code, pixel_bins)) |> 
  unnest(color_bins) |> 
  group_by(country)

Now that we have our 18 color bins for each flag, let’s look at the distribution of color percentages. Since we’re interested in the long tail, we’ll use a log scale.

flag_color_df |> 
  filter(Pct>0) |> 
  ggplot() + 
  geom_histogram(aes(Pct)) + 
  scale_x_log10() + 
  geom_vline(xintercept = 4e-03, color = 'red') +
  ggtitle("Distribution of color percentages")

We end up getting a bi-modal distribution. What are these two modes? Well the one to the right on the chart represents large color areas - the big, solid shapes of color we normally associate with flags. As that distribution tails off to the left, we get colors representing anywhere from 10% down to around 0.5%. These are mostly going to be colors that we’d be able to detect, but might represent small details of a flag like the blue shields in the Portuguese flag or the brown eagle in the Mexican flag. The second mode peaks around 0.01%. Anything representing this small of a percentage is likely to be borders or gradients, like the weird stuff we saw when we zoomed in on the Finnish flag or perhaps shades of brown gradation in the Mexican flag’s eagle that got color-binned in separate boxes from the primary brown color. Even though these colors are real, they don’t feel in the spirit of the question we’re trying to answer. Just to convince you, let’s find a random flag with a binned color representing around 0.01% and see if we can detect it.

set.seed(5446)
flag_color_df |> 
  filter(between(Pct, 9e-05, 2e-04)) |> 
  ungroup() |> 
  sample_n(1) |> 
  select(country, Pct, hex)
## # A tibble: 1 × 3
##   country       Pct hex    
##   <chr>       <dbl> <chr>  
## 1 Portugal 0.000117 #4D5D94

Pakistan is our random example. Let’s look at the flag.

Pakistan: A green and white flag

This is a good example, because this flag is clearly supposed to be just green and white. The color binning technique, however, found that 0.01% of the pixels in the image belong to a color bin distinct from the green and white color bins - a light green color that must represent the edges of the green and white areas.

Where is this?

So where do we draw the line exactly? That’s hard to answer perfectly, but based on the shape of the distribution I feel that anything in that second mode probably doesn’t deserve to be counted as a real color. Thus, I’m making the call to draw the line at 0.4%, which you can see in the red vertical line of the histogram. In the end, my initial guess of 1% as the cutoff wasn’t too far off! If anything, this essay is a tribute to human intuition.

Belize with 9 colors!

And in the category of most colors for a flag comprised of solid shapes the winner is . . .

South Africa with 6 colors!

Personally, if this was an Oscars-style award show I would have the second category be the equivalent of best picture which makes South Africa our big winner.

I love using data to better understand complex problems. What I generally find, though, and this was especially true here, is that data begs us to rephrase our initial question and use more exact language. It can be frustrating and grating. And as we iterate on our original question, refining it and making it more exact, we have to also make sure we aren’t losing the spirit of the initial query. It’s a give and take, a push and pull, an art and a science and a struggle to align cold, hard logic with the line of thinking and intelligence that our collective brains have evolved to rally around. This might be what I love most about working with data.

Further Reading

1. History of the RGB color model
2. Kahn Academy Lesson on the CIE color space