Accelerate creativity in Unity

April 27, 2020 Eric Risser

When rapid, truly disruptive progress occurs in an industry, instabilities are created – and unstable systems don’t last. We see many examples of this everywhere, but most of us rarely think about what it means and how this kind of instability might be taking place within the computer graphics industry.

Why an example-based workflow?

Computer graphics is rapidly progressing, which leads to an unstable supply and demand market. The demand for 3D content to consume has long outpaced our ability to produce it. We see many symptoms of this inequality: exponentially growing team sizes, ballooning project budgets and durations, and dependence on distributed teams and outsourcing. These are cracks in the industry’s foundation.

The rising cost of game development is exponential.

Thus, we’re making a case for a new paradigm in content creation, a shift in mindset as well as in the technology stack: the example-based workflow.

What is an example-based workflow?

The philosophy behind an example-based workflow is that we can’t keep painting and programming every pixel. Rather than starting from a blank canvas and building everything from scratch using a manual or procedural workflow, this workflow starts with an example of what’s needed, such as a scan, a photo, or an older asset from a previous generation of media. Starting with this raw example, the computer requires data, high-level instructions on what to do with that data, and then it can begin to automate an otherwise manual process.

The example-based workflow.

To make things simple, here’s a list of the three ingredients that make up an example-based workflow:

  1. Start from seed data or an example of what an artist wants to create
  2. Define the intent
  3. Automate
  4. Automate

This might sound unconventional, but the industry is already using example-based workflows today. Although it’s a niche use case, photogrammetry, the conversion of 2D photos into textured 3D models, can be considered the first wave of example-based content creation because it leverages data to automate the 3D modeling process. Unity sees AI-Assisted Artistry as the second wave.

When Unity began to explore photogrammetry and how that would fit into their platform, it was a small team on a strict timeline for this project. This necessitated adopting the most efficient methods possible – brute force wasn’t an option. Most studios see scanning as a means to create highly photorealistic content very quickly. Studios are investing in scanning equipment and associated software such as Reality Capture, which has gained an excellent reputation. That said, photogrammetry is not a full turnkey solution, since manual cleanup and tweaking are needed to go from raw scan to production-ready asset. This is where Unity had to develop and explore other solutions – leading Unity technical artist Cyril Jover to create the industry’s leading Delighting tool, designed specifically for photogrammetric scans. As part of this effort, the team adopted ArtEngine for automated seam removal and the general scan-cleaning process.

Microsoft technical artist Pete McNally is similarly charged with the task of building realistic environments with a small team. These constraints led Pete to become an industry pioneer in adopting and utilizing an example-based workflow. Pete shared his full thoughts on the topic at 80lvl:

"There are already some solutions for removing seams on materials but I haven’t found anything to match ArtEngine by Artomatix. It uses an example-based approach, rather than procedural to remove seams and can also mutate textures, creating new textures based on input from existing ones. For example, I used Artomatix in the Сraggy Cliffs:

Mutation and seam removal on photogrammetry scans courtesy of Pete McNally.

It has given a new lease of life to some 3D scans I’ve done that were too problematic to be used in production, such as textures that were too blurry or geometry that was stretched or had holes in it. ArtEngine can fill in missing detail, allowing you to paint areas that you’d like to ignore and producing new viable textures.”

So that’s what we mean by an example-based workflow: the computer extrapolates and ideates upon example data, so the artist provides high-level sophisticated creativity and the computer contributes the low-level mechanistic creativity. It’s a perfect symbiosis between artist and tool, each empowering the other. This isn’t just a necessary evolution for the computer graphics industry, it’s an inevitable one.

How does it work? Enhance, expand, transmute

You’re probably wondering how you would use an example-based workflow in practice to streamline your creative productions. We’ve identified six pillars to the workflow, three on the data side and three that automate. While each of these pillars warrants their own blog post, I’d like to briefly touch on automation by highlighting the three tasks that should no longer require manual intervention, since this is likely foreign to many artists.

Enhance

Enhance is a family of ArtEngine tools that help you to improve your visual creative assets. The industry is sitting on mountains of old, outdated 3D content. The standard resolution for textures doubles roughly every five years. At the same time, original master files are frequently lost, leaving behind only downressed and compressed production assets. Even assets produced by scanning today can suffer from blur and noise artifacts. ​Enhance takes assets and improves the quality in some way, extending their value and lifespan. One ArtEngine feature from the Enhance family is Up-Res, ​which can perform a ​2x ​or a​ 4x cartoon or photorealistic resolution improvement on any image or texture.

Expand

Going from a concept to filling an entire world is a long, tedious and largely repetitive process. Expansion was the first problem Artomatix set out to solve, and it’s still one of our core goals. We’re proud to support a number of expansion-based features in ArtEngine today. Seam Removal can make any texture or material tileable. Content Aware Fill can fill in any damaged or missing parts of a scan. ​Texture Mutation ​can grow a material out to a bigger size without creating repeats or generate infinite seamless variants from just a single example.​ Expansion brings the richness and variety of the real world into the digital realm.

Transmute

Often 3D content is unfit for the task at hand – not that there’s anything wrong with the content itself, it just doesn’t work with the rest of the project. Perhaps a material is in the wrong artistic style, or a PBR scan just doesn’t look right placed next to other scans due to scanner miscalibration. If putting together a 3D world is like assembling a puzzle, ​Transmutation ​is any function that helps the artist when those puzzle pieces don’t fit together other out of the box. Many real-world objects such as cloth and brick walls are “near-regular patterns,” this means they have a pattern, but it’s warped or inexact in some way and can’t be used as a tileable texture. The Pattern Unwarp ​ feature automatically detects the true pattern and unwarps it, transmuting ​it into a form that can be tiled.

We’ve briefly introduced the three pillars of automation, or the three ways that artists can use an example-based workflow to transform their 3D content in the future.

Watch the Unity ArtEngine session

If you’re interested in learning more, watch Artomatix founder Eric Risser and Mike Geig, Global Head of Evangelism Content, show you ArtEngine’s potential for your projects. In this free one-hour online webinar, they demonstrate how to take full advantage of an example-based workflow with the help of ArtEngine’s features such as Texture Mutation, Seam Removal, Pattern Unwarp, Material Generation, Texture Upresing, Color Match, JPEG Artifact Removal, Texture in-painting, and more.

Watch the Unity ArtEngine session

Previous Article
Artists’ best practices in Unity
Artists’ best practices in Unity

Optimize game art and improve performance when building for mobile.

Next Article
Round trip easily between Unity and Autodesk
Round trip easily between Unity and Autodesk

Pointers on efficient round-tripping of your geometry between Autodesk applications and Unity.