Cover for Color digitalization and data saving article

Introduction

A color seems complex to understand because sometimes we use CMYK values to talk about it or RGB or even Lab, XYZ, Lch, ... All those color spaces could define the same color depending of the use.

In fact the purest way to speak about a color is through its spectral data. Those data are measured directly through a spectrophotometer. Though we can have different spectral data for the same color depending of the measurement condition : without illuminant every color is black, so let's use an illuminant but which one ? Do we use the D50 normalized illuminant or the D65 normalized illuminant, should we use a filter or a polarizer (to prevent gloss interfering)? It does not seems to have one unique way to save colors.

There is a long way before a true digitalization of a color in the real world...

What does color spectral values look like

Why does accurate color digitalization is that important ?

There are two main goals of the color digitalization: saving and sharing.

Saving

As a picture would, by saving a color you can create a fixed point in the time of your color's perception (because the measured subject can change through the time by growing old, burning at the sun, ...). You can use it as a reference like Pantone does with its inks or keeping it to create a history and check the evolution over time.

Sharing

You might have laboratories that measure and work with the color within the same building but colors workflow often have many places all around the world. In this last case the manufacturer may never see the color reference he is about to reproduce, so he does need a way to know what he is supposed to do and what does it looks like.

There are others cases but let's keep it simple for today.

Current saving system

There are many way to save color information, indeed some well known companies such as X-Rite or Data Color though about it and made their own formats.

But all of them are very different from each other and make complicated the compatibility between software. This issue was a starting point of the Coraye software : creating a bridge between software by implementing import and export of any kind of files (e.g.: allowing to transform a QTX file to a CxF one). To achieve this, we needed to create a way to become compliant to any kind of file format and keeping data untouched (no rounding or transformation process) by structuring data.

Our first though was to choose one existing file format as a reference for all other one but we did not found the perfect one because our requirement for the Coraye Software are too specifics. Moreover we choose to work with web technologies and encoding such as JSON format whereas standardized files are often proprietary oriented and complicated to parse.

The QTX format

Created by DataColor, the QTX file store data in a way that each color is independent from each other. So you will find informations such as the spectrophotometer used or the measured date for each sample.

You do have the possibility to insert batches colors. Those are repeated measure in different conditions (e.g.: with a different spectrophotometer).

You can even set different custom keys with custom value without worry. This would allow the file system to grow without requiring a v2 format that would not be compatible with previous versions.

[STANDARD_DATA 0]
STD_NAME=11-0605 TCX
STD_GUID=3B5F4330-4B43-715F-4146-BB2B4146BB2B
STD_DATETIME=1304360181,
STD_REFLPOINTS=36,
STD_REFLINTERVAL=10,
STD_REFLLOW=400,
STD_VIEWING=%R LAV  SCI UV 400
STD_X=76.039398,
STD_Y=79.696068,
STD_Z=79.106171,
STD_R=68.957001,71.864998,72.008003,72.107002,72.460999,73.196999,74.137001,75.147003,75.987000,76.930000,77.723000,78.228996,78.500999,78.888000,79.075996,78.955002,78.995003,79.988998,81.500000,82.607002,83.119003,83.142998,83.023003,83.170998,83.910004,85.121002,86.570999,87.936996,89.249001,90.633003,91.738998,92.385002,92.737000,92.898003,92.920998,92.829002,
STD_MEASDLL_PARAMS=Manufacture:unknown,Model:CE7000A,Serial number:unknown,FirmwareVersion:unknown,Geometry:d/8,Specular component:I,MeasSpot:N/A,MeasArea:LAV ,UV-Filter%:UVEXC ,UV-Cutoff:None,MeasurementType:R,MeasurementSource:Instrument,Number of bands:36,Start band:400,Bandwidth:10,Correlation:Off,
STD_INST_TYPE=CE7000A
DCC_JOB_COMMENT=
Vendor Name=
Vendor Number=
Mill=
Dyehouse=
Vendor Email=
Retailer Name=
Brand=
Division=
Department=
Season=
Delivery Number=
Retailer Email=
Color Name=
Color Number=
Fiber=
Fabric Style=
PID=
Submit Number=
Labdip_Number=
Date Submitted=
User Name=
PRG_PALETTE_PROGRAM=
PRG_COLOR_PROGRAM=
PRG_COORD_PROGRAM=
PRG_DATE_DUE=
PRG_DATE_SUBMIT=
PRG_DATE_REVIEW=
PRG_ITEM_STATUS=
PRG_ITEM_TYPE=
PRG_JOB_IDENTIFIER=
PRG_FORMAT_FILE=
Single color encoding in a QTX file
Pro:
- You can mix colors in a same file
- You can customize the content
- Human readable
Cons:
- Not standards format (Require custom interpreter)
- Repetitive content
- Custom key mean specific code to interpret (data is lost if the software that import the file is not able to work with it because it doesn't understand what PRG_DATE_REVIEW=XXXXX means)
- Full of unused empty keys (more difficult to read even for humans) that make file heavier that it should be
- Difficulties to use with basic tools such as XCEL

The Cxf format

Created by XRite, the Cxf file is a complexe format that allow you to store various information from Lab to spectral, with metadata. It's a complete format that allow a full customization. But what seems to be a key functionality could also be a dangerous one : the over-customization tend to make difficult to read what's essential.

More over the XML format also have a lot of repetitive content. You can escape this by settings common parameter but it does only make the file even more complicated to read.

<?xml version="1.0" encoding="utf-8"?>
<cc:CxF xmlns:cc="http://colorexchangeformat.com/CxF3-core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <cc:FileInformation>
    <cc:Creator>Coraye</cc:Creator>
    <cc:CreationDate>2018-07-16T19:30:53.225Z</cc:CreationDate>
    <cc:Description/>
  </cc:FileInformation>
  <cc:Resources>
    <cc:ObjectCollection>
      <cc:Object ObjectType="Standard" Name="Orange" Id="C0">
        <cc:ColorValues>
          <cc:ColorCIELab ColorSpecification="S0">
            <cc:L>90</cc:L>
            <cc:A>78</cc:A>
            <cc:B>125</cc:B>
          </cc:ColorCIELab>
          <cc:ColorSRGB ColorSpecification="S0">
            <cc:R>255</cc:R>
            <cc:G>151</cc:G>
            <cc:B>0</cc:B>
          </cc:ColorSRGB>
        </cc:ColorValues>
        <cc:PhysicalAttributes>
          <cc:Quantity Units="TINT PERCENT">100</cc:Quantity>
          <cc:Opacity>100</cc:Opacity>
        </cc:PhysicalAttributes>
      </cc:Object>
    </cc:ObjectCollection>
    <cc:ColorSpecificationCollection>
      <cc:ColorSpecification Id="S0">
        <cc:MeasurementSpec>
          <cc:MeasurementType>Colorimetric_Reflectance</cc:MeasurementType>
          <cc:GeometryChoice>
            <cc:SingleAngle>
              <cc:SingleAngleConfiguration>Annular</cc:SingleAngleConfiguration>
              <cc:IlluminationAngle>45</cc:IlluminationAngle>
              <cc:MeasurementAngle>0.0</cc:MeasurementAngle>
            </cc:SingleAngle>
          </cc:GeometryChoice>
        </cc:MeasurementSpec>
      </cc:ColorSpecification>
    </cc:ColorSpecificationCollection>
  </cc:Resources>
</cc:CxF>
Single color in CxF format
Pro:
- Certified file
- Standardized way to communicate color
- Possibility of customization
Cons:
- Does have a versioning that require a program to be up to date with the latest rules
- Chaotic over-customization
- The data's path is not always intuitive (where should I write spectral data, Resources>ObjectCollection>Object>ColorValues>ColorCIELab to get Lab values)

The CGATS format

This format is not used as a color table format, it's mainly a file used for printing targets but it can be considered as it (since strictly speaking it does contain similar information than you can find in a QTX or a Cxf format).

The format is quite convenient for a list of color patches that share the same condition (same support, same spectrophotometer info, same color index value). It is easy to read for a human or a computer and is even adapted for a copy paste in an excel sheet. You can add custom keys and there aren't that much repetitive content.

Unfortunately it isn't convenient for color table saving since all colors must share a common context and color values (XYZ & LAB & SPECTRAL), if one were missing, the full file would collapse.

CGATS.17
ORIGINATOR	"Fiery Printer Profiler5.1.1.16"
FILE_DESCRIPTOR	"Output Characterisation"
CREATED	"2019-11-04 15:48:00"
INSTRUMENTATION	"X-Rite i1iO/i1iO2"
KEYWORD	INSTRUMENTATION_SN
INSTRUMENTATION_SN	""
KEYWORD	INSTRUMENT_FILTER
INSTRUMENT_FILTER	"M2 - UV cut"
KEYWORD	TRACKING_ID
TRACKING_ID	"7865-78T6"
KEYWORD	LAYOUT
LAYOUT	"1847 patches"
KEYWORD	MEASURED
MEASURED	"2019-11-04 16:12:53"
BEGIN_DATA_FORMAT
SAMPLE_ID	CMYK_C	CMYK_M	CMYK_Y	CMYK_K	XYZ_X	XYZ_Y	XYZ_Z	SPECTRAL_380	SPECTRAL_390	SPECTRAL_400	SPECTRAL_410	SPECTRAL_420	SPECTRAL_430	SPECTRAL_440	SPECTRAL_450	SPECTRAL_460	SPECTRAL_470	SPECTRAL_480	SPECTRAL_490	SPECTRAL_500	SPECTRAL_510	SPECTRAL_520	SPECTRAL_530	SPECTRAL_540	SPECTRAL_550	SPECTRAL_560	SPECTRAL_570	SPECTRAL_580	SPECTRAL_590	SPECTRAL_600	SPECTRAL_610	SPECTRAL_620	SPECTRAL_630	SPECTRAL_640	SPECTRAL_650	SPECTRAL_660	SPECTRAL_670	SPECTRAL_680	SPECTRAL_690	SPECTRAL_700	SPECTRAL_710	SPECTRAL_720	SPECTRAL_730
END_DATA_FORMAT
BEGIN_DATA
1	60.00	0.00	50.00	0.00	41.64	60.43	43.78	0.14773	0.17724	0.19816	0.23544	0.30003	0.36096	0.41560	0.47331	0.54287	0.62900	0.70874	0.75830	0.78469	0.79415	0.79218	0.78167	0.76566	0.73601	0.68953	0.63560	0.56756	0.48263	0.37687	0.27670	0.20821	0.16947	0.14420	0.12414	0.11660	0.12573	0.14979	0.18434	0.22296	0.24888	0.25621	0.26340
2	10.00	100.00	20.00	0.00	35.70	18.06	20.42	0.22300	0.27595	0.31391	0.34810	0.38639	0.38871	0.35439	0.29652	0.23688	0.17549	0.11434	0.07549	0.05133	0.02626	0.01216	0.01064	0.01226	0.01059	0.01216	0.04120	0.16407	0.36062	0.51761	0.59821	0.62715	0.64484	0.67501	0.71278	0.74878	0.77579	0.79784	0.81395	0.82720	0.83588	0.84080	0.84354
The first two colors of a color table in CGATS
Pro:
- Easily readable
- No repetitive content
- Can store spectral data with custom wavelength index
- Easy to export to tool such as XCEL
Cons:
- Colors must share a common context
- Each color have the same data structure
- Text format is not convenient for machine parsing

Other formats

There are also other file formats that are made for color saving but those aren't efficient because they unfortunately do not support spectral data. It can mainly be explained by the fact that those file format are made to save spot colors only and are more focused on graphic art than printing.

Examples of other formats : Ase, Acb, Aco, Csv, Coreldraw, ...


Let's start structuring color data !

Now that we've seen other formats and understand what are expectation among all of them, let's build a way to save our colors.

Has previously said, we will use the JSON format, in that way it will be easier to parse for an online program or an online database.

Here is the start of our color file :

{}
empty JSON file

Color requirements

If we want to ensure compatibility with QTX files, we need equivalent for following keys :

  • STD_NAME (Name of the color)
  • STD_DATETIME (Created/Measured date)
  • STD_REFLPOINTS + STD_REFLINTERVAL + STD_REFLLOW + STD_R (Spectral values)

If we want to add the CxF compatibility let's add equivalent for:

  • Name (Name of the color - Common with QTX one)
  • ID (Identifier within the file)
  • ObjectType (Standard, Trial, Target, Substrate, Colorant, etc.)
  • GUID (Identifier unique worldwide)
  • Creation Date (Created/Measured date - Common with QTX one)
{
  "id": "CN1",
  "guid": "CORAYE-CN1",
  "name": "My Color Name 1",
  "created_date": "Thu May 07 2020 17:47:46 GMT+0200",
  "color_preview": [0, 0, 0], // Always RGB
  "type": "spot", // light, display
  "header": {
    "filter": "M0",
    "specular_component_mode": "SCE"
  },
  "spectrum": {
    "380": 0,
    "385": 0,
    "390": 0,
    "385": 0,
    ...
  },
  "cie": {
    "lab": [0, 0, 0],
    "xyz": [0, 0, 0],
    "observer": 2,
    "white_reference": {
      "name": "My Custom Illuminant", // Or reserved name "D50", "D65", "A", "B", "C", "E", ...
      "color_preview": [255, 255, 255],
      "type": "light",
      "cie": {
        "lab": [100, 0, 0],
        "xyz": [1, 1, 1]
      },
      "spectrum": {
        "380": 0,
        "385": 0,
        "390": 0,
        "385": 0,
        ...
      }
    }
  }
}
Color file that would support basic requirements

From this we do support QTX, CxF and CGATS format. They would work but we would still loss any other kind of information. The way we store spectral data give us every information we need : we could eventually store spectral data with irregular step.

ArgyllCMS does provide spectral curve with a 3-3-4 steps such as 380, 383, 386, 390, 393, 396, 400, ...

Adding color spaces

Every software does not include a spectral converter to get Lab values, so it would be nice to save them as well BUT to get Lab values (or XYZ ones) we require a white reference and an observer angle value. The perfect thing would be to include full white reference object but that's a little overkill for a true use of the color. But let at least give the possibility to it by specifying the white reference with a reserved name (among standards).

{
  "white_reference": {
    "name": "D50"
  }
}
Only specifying the name

Or with a detailed white reference (tip: that is the exact same structure than a color but with a light type):

{
  ...
  "white_reference": {
    "name": "My Custom Illuminant",
    "color_preview": [255, 255, 255],
    "type": "light",
    "cie": {
      "lab": [100, 0, 0],
      "xyz": [1, 1, 1]
    },
    "spectrum": {
      "380": 0,
      "385": 0,
      "390": 0,
      "385": 0,
      ...
    }
  }
}
Filling with a custom illuminant white reference

Let's merge the white reference with our main color object to get a more specific way to save it

{
  "id": "CN1",
  "guid": "CORAYE-CN1",
  "name": "My Color Name 1",
  "created_date": "Thu May 07 2020 17:47:46 GMT+0200",
  "color_preview": [0, 0, 0], // Always RGB
  "type": "spot", // light, display
  "header": {
    "filter": "M0",
    "specular_component_mode": "SCE"
  },
  "spectrum": {
    "380": 0,
    "385": 0,
    "390": 0,
    "385": 0,
    ...
  },
  "cie": {
    "lab": [0, 0, 0],
    "xyz": [0, 0, 0],
    "observer": 2,
    "white_reference": {
      "name": "My Custom Illuminant",
      "color_preview": [255, 255, 255],
      "type": "light",
      "cie": {
        "lab": [100, 0, 0],
        "xyz": [1, 1, 1]
      },
      "spectrum": {
        "380": 0,
        "385": 0,
        "390": 0,
        "385": 0,
        ...
      }
    }
  }
}
We already have a good base to prevent data loss

From this point we do have a color with enough information for a RIP or many other softwares but you might want to get more color space values of your measured color. Let's add them inside a colorspace field.

{
  ...
  "colorspace": {
    "rgb": [0, 0, 0],
    "cmyn": [0, 0, 0, 0],
    "6clr": [0, 0, 0, 0, 0, 0]
  }
}
Include all converted values in the colorspace object

Color encoding  ?

How should we encode our color values ? This is a complicated question since RGB is from 0 to 255, XYZ is from 0 to 1, Lab is 0 to 100 for L value, -125 to 125 for a and b values but every thing else is from 0 to 100. We may even need to encode a CMYKOG (O: Orange, G: Green) values.

The best is still to keep the data untouched and so saving data colors "as it": no rounding or mathematical formula required.

{
  "colorspace": {
    "rgb": [255, 255, 255]
  }
}
RGB is 0 - 255 and CMYK or even GREY 0 - 100. 

Basically, everything that is not RGB (or Lab, XYZ, Lch ...) is CMYK so it should be encoded from 0 to 100.

Where is the profile info used for conversion ?

Since the profile has already been used to get the RGB, CMYK or NCLR (N -> 1, 2, 3, ... color channel) color value, we do not need it anymore so it is not require to specify it again. But we can still add the information if you want into a metadata field. This field is made to store whatever information you may use to include context.

{
  "metadata": { // Whatever you want, this is a custom field
    "comment": "",
    "rgb_profile": "ADOBE98",
    "spectrophotometer": "MYIRO",
    "manufacturer": "Konica Minolta",
    "serial_number": null,
    "firmware_version": null,
    "area": "LAV",
    "source": "instrument",
    "correlation": false,
    "physical_attributes": {
      "dimensions": null,
      "quantity": null,
      "weight": null,
      "thickness": null,
      "gloss": null,
      "finish": null, // coated, matte, glossy, ...
      "substrate": null, // wood, paper, textile, ...
      "image": null, // binary or URI type
      "opacity": null,
      "attributes": null
    }
  }
}
Example of metadata context based on what CxF would suggest

Batches colors

The spectral data is the best way to communicate color without data loss, because you get Lab, XYZ and every other color space value from it. But the same color could have different spectral data (and so another Lab, XYZ, etc values) if you change your spectrophotometer configuration (Filter and specular component mode). That's what batches field is for : saving measurements of the same color but in another configuration mode.

{
  ...
  "batches": [
    {
      "color_preview": [0, 0, 0],
      "header": {
        "filter": "M1",
        "specular_component_mode": "SCE"
      },
      "spectrum": {
        "380": 0,
        "385": 0,
        "390": 0,
        "385": 0,
        ...
      },
      "cie": {
        "lab": [0, 0, 0],
        "xyz": [0, 0, 0],
        "observer": 2,
        "white_reference": {
          "name": "D50"
        }
      }
    }
  ]
}
Adding a batch measurement with another filter : M1

The batch data should follow the same structure than the main color but should not include another batches field into it, only the main color should have one. Everything included into the batch data should be impacted data by the filter change, every thing else is inherited from the main color.

Final example

If we include every thing that has been explained until here, we should have a file that looks like this:

{
  "id": "CN1",
  "guid": "CORAYE-CN1",
  "name": "My Color Name 1",
  "created_date": "Thu May 07 2020 17:47:46 GMT+0200",
  "color_preview": [0, 0, 0],
  "type": "spot",
  "header": {
    "filter": "M0",
    "specular_component_mode": "SCE"
  },
  "spectrum": {
    "380": 0,
    "385": 0,
    "390": 0,
    "385": 0,
    ...
  },
  "cie": {
    "lab": [0, 0, 0],
    "xyz": [0, 0, 0],
    "observer": 2,
    "white_reference": {
      "name": "My Custom Illuminant",
      "color_preview": [255, 255, 255],
      "type": "light",
      "cie": {
        "lab": [100, 0, 0],
        "xyz": [1, 1, 1]
      },
      "spectrum": {
        "380": 0,
        "385": 0,
        "390": 0,
        "385": 0,
        ...
      }
    }
  },
  "colorspace": {
    "rgb": [0, 0, 0],
    "cmyn": [0, 0, 0, 0],
    "6clr": [0, 0, 0, 0, 0, 0]
  },
  "metadata": {
    "comment": "",
    "spectrophotometer": "MYIRO",
    "manufacturer": "Konica Minolta",
    "serial_number": null,
    "firmware_version": null,
    "area": "LAV",
    "source": "instrument",
    "correlation": false,
    "physical_attributes": {
      "dimensions": null,
      "quantity": null,
      "weight": null,
      "thickness": null,
      "gloss": null,
      "finish": null,
      "substrate": null,
      "image": null,
      "opacity": null,
      "attributes": null
    }
  },
  "batches": [
    {
      "color_preview": [0, 0, 0],
      "header": {
        "filter": "M1",
        "specular_component_mode": "SCE"
      },
      "spectrum": {
        "380": 0,
        "385": 0,
        "390": 0,
        "385": 0,
        ...
      },
      "cie": {
        "lab": [0, 0, 0],
        "xyz": [0, 0, 0],
        "observer": 2,
        "white_reference": {
          "name": "D50"
        }
      }
    }
  ]
}
Final color structure

This file structure might not be perfect but tend to be as complete as possible. It mixe the use that can be made of data and takes into account colorimetric issues of the printing and graphic world. It is web focused and the JSON format is among one of the most used in the world because it is easily readable for both humans and computers.

Share:

Recents posts