A few years ago I was part of a small team of freelancers, tasked with helping LEGO transition their Boxart and Imaging from Photography to CGI.

My part in this was developing the shader system for all materials used at LEGO and helping with Code solutions to automatically assign and configure the shaders based on Context.
Lastly,I had to maintain and vet the Materials being used for final rendering, adding new ones as required.

In this Blog Post I will give details about the process and motivations.

LEGO has been around for decades and in the past all Images you've seen of LEGO have usually been Photos.

The LEGO group would publish up to 75.000 Photos per year, for example for 

• Images on the Boxes (front, side etc.)
• Images in Building Instructions
• Posters for Stores and promotional Material
• Images for online Sales Sites (Amazon, LEGO.com etc.)
• Toy Fairs and Partner Sales events (e.g. presentations to big Toy Chains for them to know what's coming)
• Prototype Development and evaluation

Having to deal with these amounts of Photos was/is quite resource-hungry.
Not only do you have to assemble all the LEGO sets, you have to pose and light them in a studio, transport them and process and clean the final photos (e.g. to remove oil, smudges, dust and Fingerprints)

Example of amount of pictures associated to just a single LEGO Set on LEGO.com

Apart from this, we live in a digital world .. and while LEGOs core business is selling bricks, they need to deliver digital assets to many different partners and for many different purposes like

• building instruction steps & digital (animated) building instructions
• animations and turntables of sets for online retailers and LEGO.com
• delivery of brick assets to outsource vendors e.g. Film and Games Companies
• Digital assets can be easily used in Machine Learning applications e.g .to identify bricks from a photo and spit out ideas, what you can assemble with the bricks in the photo

While our mission was to create photorealistic renders of LEGO, it became clear quickly, that there were different opinions about what that meant.

A long standing joke was, that while we in the Box Art Team tried to make things look more photographic, the department processing LEGO images was trying to make the products look more like CGI, having removed and retouched surface imperfections in actual Photos for decades.

As such there was always a balance to be found, in what Marketing wanted and what the CG division tried to achieve.

Example of common retouch work done from a photo to final printed output

In our transition to CGI boxart we didn't have to start from zero. 

LEGO has in fact for a while created digital models of their Bricks derived from production CAD data.

These models are/were in use for

• Creating the "Steps" in the Building Instruction Booklets
• used to build and prototype sets within LEGOs proprietary LDD Software (LEGO Digital Designer) 

The LEGO Digital Designer is available for download for anyone, though the free version is no longer maintained and differs quite a lot from LEGOs inhouse version “LDD Pro”

The free version of LDD and LEGO's more advanced LDD Pro

While the LEGO Digital Designer is a powerful tool, it lacks proper rendering and cannot compete in that regard with standard DCCs like Maya.
However it did mean we had some data to rely on when building sets in Maya.

Using a set of custom Maya Scripts developed by the CG Boxart Team  ..

• ASCII/XML files exported from LDD were parsed to get brick IDs and position
• Digital versions of Bricks were loaded from LEGOs Digital Database and placed and grouped accordingly
• "Common Parts" (e.g. the knobs) were replaced with standardized common parts we managed ourselves.
• Materials and Decoration Prints were applied based on “Recipe” Data sourced from LEGOs production database
• "Stickers" were positioned and "bend" onto the underlying surface.
• Rigging Controls were added to pose models with correct connectivity data.
• Bricks were randomly rotated in very small increments to simulate build imperfections

The result of a Model Build in Maya.
In an ideal scenario very little manual editing should be required after apart from posing and lighting.

Though of course this was the end result. Getting to that point took a while.

Making sure Materials and Colors are correct - that was my job while the others dealt with Model Build, Building Instructions etc.

Having been a professional Texture and Lookdev Artist for quite a lot of years, I was admittedly naïve how complex this task is.

After all, it's just plastic right ? Wrong.

LEGO uses more than 200 different plastic types and mixtures. Each one can alter the look of a brick.

Below are three examples of how for example the translucency changes depending on what Plastic Type is used

Example how different plastic types can affect the translucency of the material

In addition to the pure plastics, there are also a surprisingly large amount of other materials in use e.g.

• Rubbers (with different mixtures making them harder or softer)
• different Foils 
• Stickers
• Strings
• various Fabrics such as cotton, nylon etc.
• Metallic print applications on Bricks, Fabrics and Foils
• Special Effects (Glow in the Dark, "invisible Ink", UV Light reactive, Glitter, Opalescence etc.)

Did you know that LEGO is the world's largest tire manufacturer ? Small ones for sure but never the less :)

They also use a lot of fabrics

Each Lego color has an ID Number.

Over the decades the number of color IDs has reached hundreds, however only a small subset of colors is "active" at any time - meaning they are colors that are going to appear in a year's products. 

Example of a subset of colors with their unique ID Number

As you can see above, a color ID means more than just the pure color. It can also define unique surface properties

• 114 is transparent and has embedded glitters
• 312 is solid
• 182 is transparent
• 297 has a gold coating
• 364 is transparent and has an opalescence/iridescence finish

This introduces certain problems when trying to create a modular material system, since just knowing the raw plastic type for a brick is not enough to adequately define surface properties

When loading in a brick into Maya,we could rely on a unique recipe for each brick for that particular LEGO set, that was obtained directly from the LEGO production database. This gave us the following information to construct a surface

• what the Raw Material Number of the Brick or Piece (e.g. Fabric type, Plastic Type) was.
• what the color ID of the Brick or Piece was
• what print should go onto the brick and what UV set the print should be assigned to.
  Do determine exact print position many different standardized UV sets were available, making it relatively straight forward to get an accurate position of a print on any given model

Each Shader had certain Node names that the build scripts would look for.

• A JSON Data file was used to read in per-color ID data and configure base colors but also individual color-id specific behaviors such as activating glitter, changing the subsurface color etc.
• Layered Textures Nodes with fixed Naming were pre-authored where the Build Tools could automatically add File Textures to, containing the Element Prints (think for example the Face Print on a minifigure)
• Using User Attributes on each Brick, individual Roughness Zones (so called VDIs) could be defined
• Using User Attributes on each Brick, additional data could be added per brick (e.g. Serial Number Bump Maps etc.)

Example of an early JSON file prototype configuring Shaders based on Color IDs in the Element Recipes.
Later on these JSON files would hold several values per Color ID, configuring for example Subsurface strength and colors.

Let's just say LEGO is pretty anal about its Colors and how they are represented. 

Ensuring correct colors and color behaviors was therefore high on the list of things to achieve while being a constant game of "chase your own tail" when new colors were introduced. 
CGI can always just be an abstraction of the real thing though, and especially at the tiny scale of a LEGO element, a smallshift in value can make a large difference.

As a first step I lookdev'd all active LEGO brick Colors for correct specular, diffuse color and subsurface.

I also created the various base shaders, for example for different plastic types and/or configured the JSON files to auto change shader settings as required,based on Color ID and Raw Material

Below you can see the same Color ID (40 - fully transparent) on different plastic types, while each Plastic Type was a separate Base shader

Example of different Rubber Mixtures (Raw Material is often SEBS), which I simplified down to just have one shader for "Soft Rubber", one for "Medium Rubber" and one for "Hard Rubber"

As was the case for brick colors, Rubbers could have different colors as well (here the little cone element), that needed to be set up via shader attributes and our color configuration JSON Files

Printed elements come in two flavors

• directly printed onto the element
• applied as a Sticker by the "Builder / Player"

Example of a direct print on a Minifig, a sticker sheet & and a Sticker glued onto a brick

As previously mentioned direct prints (called "Decorations") can be applied to fixed positions on a LEGO element, something that was implemented with UV sets in the CGI Box Art workflow e.g.

• the decoration file was sourced from LEGOs digital Production database, then automatically applied to a file specific UV set in the shader. This UV set workflow was inherited from LEGO Digital Designer, which allowed use of existing data structures.
• Slots were exposed in the shaders to attach (vray) file nodes to, and the deco specific UVset target name was added in the file node, connecting the image to the uv set.

Direct prints in real life are performed using a sponge printing technique, which leaves characteristic irregular patterns at the edges, bleeds the colors slightly as well as inherits some of the underlying brick structure (e.g. crevices might not get caught by the ink).

Below you can see a closeup of a direct print (photo).

Apart from the mentioned irregularities, also notice the layering in the print.
The white was printed on first, other colors followed. Subsequently some spots remain white even in the neighboring colors, where the sponge did not reach.

To simulate this effect we used Substance Designer to pre-process the perfect Vector illustrations of decorations to get a more natural printed look. This gave us the following look:

Stickers were mostly straightforward - at least from a shader point of view.

A large percentage of stickers is created from the same base material, meaning I could author a generic sticker shader and using the same system used for bricks, expose certain nodes in the shader to accept dynamic image inputs for the actual image

Example of generic sticker shader fed with different stickers with or without alpha

Of course 'applying" the stickers correctly in the digital Model build was less straightforward, as it involved quite some gymnastics to know which sticker should go onto which brick - and also the pure technical challenge of accurately - and automatically - bending the stickers onto the surface underneath.
But that was not my task :) 

A more complicated aspect of prints - both direct element prints and stickers - was the existence of metallic prints, which needed special shader settings to look correct.

Below you can see an example of 3 different metallic effects in prints (gold, "laquered gold" and silver)

Luckily when LEGO authors decorations and stickers, they utilize a fixed color ID system (using the same color IDs as for Brick Colors). That meant that metallic areas in the raw decoration and print files conformed more or less to certain RGB Ranges.

To achieve the metallic prints without any further human interaction, the shaders for all elements (bricks, stickers, fabrics etc) were authored in a way to automatically apply metallic effects if needed.

I created a custom OSL Node that was fed with the (dynamically) inserted Decoration and Sticker files we already ingested into the shaders. 

The OSL file would create masks based on predefined RGB Value Ranges and output masks for the different identified zones, meaning I could pre-author effects for gold, silver. bronze prints etc. that would dynamically turn on when required by the input image.

A custom OSL node was used to detect Metallic Ranges of Decorations in the shader and create masks for metallic shader effects

Below you can see the decoration shader with and without the metallic shader effect, as well as the Masks generated by the OSL node.
The fourth image was an automated validation (more on that later) if metallic ranges were picked up correctly in the shaders.

You might not have heard the term "VDI" before but I can guarantee you have seen its effect.

Matter of fact the mouse (or wacom pen) in your hand as well as the Keyboard under your fingers have VDIs.

VDIs are an industrial standard for surface finishes.

VDIs on LEGO Bricks serve different purposes

• Aestetics
• Roughing up the surface so it releases easier from the brick mold
• production method of a part of a brick to remove excess material (High Speed Milled or Grinded)

Different VDIs on different Brick Elements

LEGO uses mostly only a handful of different Surface finishes like

• VDI 3400 / 16
• VDI 3400 / 21
• VDI 3400 / 24
• VDI 3400 / 27
• VDI 3400 / 30
• VDI 3400 / 33
• Grinded (Production Method)
• High Speed Milled (Production Method)
• Diamond Polish (Production Method)
• Lasered (Production Method)

Example of different Surface Finishes

The first texturing & shading task was simply recreating the individual Surface finishes - not worrying too much about how or where to apply it.

This meant studying the surface responses in depth and highlights.
Notice for example how the highlights change from round to elongated (anisotropic) on the grinded surface

The basic surface property maps (e.g. height maps and roughness map) were created using a mixture of scans and Substance Designer, then inserted into the shaders as different slots that could be activated one by one
The shaders were balanced to mimic the effect and to cheat certain things (e.g. to avoid excessive filtering of the tiny details in the maps)

Different height maps of VDIs

Digital recreation of the VDIs on the Surface Fan and on an example Brick Element

To drive the VDIs User Attributes per Brick, I exposed Vray User Attributes in the shaders, that were auto-connected to per-mesh when the Maya Build Process ran.

Simple Sliders on each mesh would allow then to

• turn on a Surface Finish globally for the entire element
• apply texture maps to it

The same system was also used to expose additional overrides e.g. to change shader roughness (for specular control by the Lighting artist), and apply additional per-brick bump maps (for example for additional details such as mold lines, serial numbers etc.)

Example showing per element override attributes, activating a VDI Surface Finish for a brick

Lookdev'ing the Surface Finishes &  modifying the Build Tools to automatically add attributes and be able to drive them on a per brick level was only half the equation.

Remember, LEGO has thousands of Bricks and each one has not only different VDIs per area but also additional details we want to capture such as serial numbers and mold lines.

Below you can see a relatively simple Minifigure Arm, with already 4 different Surface Finish Zones

VDI Information on Bricks is usually added as Colors to the CAD Data, alongside an PDF File that is supplied to the Factory.

Unfortunately the Colors are not necessarily identical. While for example "Blue" is used to define VDI 24, the exact blue values differ, making automatisms difficult.

Before diving really into per-brick details I wanted to first cover ALL Bricks in Lego's Library with the Surface Finish that covered the largest area. I dubbed this the "Signature Surface Treatment".

The initial goal was having a "Signature VDI" for each Brick in LEGOs Library, stored again in a JSON File, that the Build Scripts could read out and transfer onto the Vray User Attributes - therefore activating a global VDI per Brick.

I wrote a maya toolset to help with VDI Authoring, to speed up this process. The tool was initially designed to quickly get the necessary information for a brick (e.g. a sheet like the one above, or the CAD data) then assign and save a "Signature VDI" to a JSON File.

The VDI Authoring Tool was a Helper Tooll I developed that was n heavy use to author and edit VDI Data for elements in the library

However while this tool allowed me to quickly get the information per brick,I would still have to manually look at all CAD Data (thousands) or Surface Finish PDFs and assign the most prominent VDI via the tool to the JSON File Database.

So further automation was required.

I wrote an auto-processing that could be fed with brick IDs (or the entire Library of elements). The automation would ..

• load in the CAD data into Maya for a given Element ID
• determine the largest "VDI Color" Area on the Geometry
• Look up the color against a database of colors to figure out which VDI Color it is
• Write the VDI Information into our "Signature VDI" JSON file associated to the Brick ID
• Do a small validation render with the detected color

Using this system I could process all Signature VDIs for all Brick elements in about 12 hours, while having to rerun it a few times to finetune the color detection to account for unusual RGB Ranges (where the element designer eyeballed colors).

Having this system also meant that new Brick Elements could be easily processed by re-running the tool once per quarter.

While the "Signature Surface Finish" that was auto-applied per birck on build (onto our Attribute Overrides) gave us a lot to work with, it was clear from the beginning that this alone would not suffice - and introduce issues.

Some of LEGOs most iconic "rough" bricks are the so called "roof tile bricks", which are overall smooth but have one slope covered with a very rough (VDI 33) surface finish.

A roof tile brick with one slope having VDI 33

With my "Signature VDI" technique best case the surface would be entirely smooth (as smooth is the largest surface area), but on other bricks worst case the high VDI is applied everywhere. Which would be totally wrong.

So I always knew that once we had the basics covered I would have to bite the bullet and start creating individual textures for each brick element, capturing accurate Surface Finishes. We were also required to include additional details not present in the models - such as Serial Numbers, Mold lines, LEGO Logos etc. .. all of which can have quite the influence on the look of a brick

Example of clearly visible serial numbers, really changing the look for example on refractive materials, tire sides etc.

The Brick Map creation could only be automated a little - mostly it was good old hands-down work in The Foundry's Texture Application MARI, with about 40 bricks done per day.

Steps that I could automate were

• Auto aligning of CAD Data to Brick Model and baking the CAD Color data onto the Brick
• adding tools to MARI to help with the process.
• Creating Mari Paint Through Images to help with Serial Numbers, Mold Lines etc.

But nevertheless it was still quite manual, and my colleague Pim Hendricks and myself managed to get through about a thousand bricks, with about 1 day per week allocated for Brick Maps

Below you can see some screenshot examples of authoring Brick Maps in Mari.

The process was a mixture of cleaning up the bakes (the transfer of CAD VDI Colors onto the lowres Model), making sure the color ranges were accurate for what the shader expects as well as splitting out and/or adding Serial Numbers, Mold Lines, Injection Marks etc.

Some examples of Mold Lines and Injection Marks (circular shape) added as Bump Maps

All Texture Maps were exported into separate folders corresponding to Brick IDs.
If a Brick ID had a set of Texture Maps, those texture maps would then on build be applied to the Attribute Overrides in Maya

Let's take a look at some examples of VDIs and Brickmaps in action

The final piece of the puzzle of creating materials and shaders for LEGO's new digital Boxart was a solid Validation and Vetting system.

Every change in the library such as new shader settings, color changes etc. had to be tested thoroughly across different scenarios before being rolled into production of on-the-shelves boxes.

To help with this I wrote a system to easily submit large amounts of validations. The system would

• Run build scripts using the latest shaders and colors on pre-defined scenes - for example to test all available colors with alll available shaders
• Run a series of tech checks to check if shader features work as expected
• Run test builds and renders of different LEGO sets

The tool would build all scenes, submit renders to our Deadline server, and after the renders were done, add information burn ins (the little boxes in the corners) to the images. Finally all the images were assembled into easily viewable slide show movies.

Rerunning the validations for each library update gave us a good history to compare changes across a wide variety of objects and LEGO sets

Example of Color ID Validations. Each Color ID would be rendered with all available base shaders in multiple light setups

Examples of validating different plastic types (more tansparent, more foggy etc.)

Generic Minifig Checks

The below were tech checks for Metallic/Non Metallic printed Decorations.

1, 2 ) The first two images were examples of Decorations containing metallic printed areas

3) Different metallic value ranges for decorations on a simple Plastic sheet to test OSL Nodes

4) A reverse test of non-metallic Decorations on metallic Color IDs.

Some different rubber mixtures (soft, medium, hard) with different surface finishes

Examples of some validation sets

The Boxart pipeline is likely a continued endeavor for LEGO however during my time we definitely made some good first steps.

Several hundred Boxes with CGI Images were shipped. Below are some examples