Utilizing Gen AI for Procedural Content Generation in video games

As the credits rolled on my TV screen, a feeling of emptiness set in. A five year wait for this beautiful game was done in an eight-hour binge & I simply yearned for the next season to arrive superfast. Game developers have long relied on procedural content generation (PCG) to accelerate game development and minimize manual effort, yet it takes years to develop a great game.  Being in this industry, I understand that each masterpiece is a symphony of passion, love, and dedication, where game developers pour their souls into crafting every pixel and line of code to tell a story never told before.  But I couldn’t help thinking if Generative AI (Gen AI) is the silver bullet to churn out masterpieces quicker & faster?

PCG involves algorithmically creating various game elements like levels, maps, items, characters, and mechanics instead of handcrafting them. The evolution of PCG can be delineated into three key phases:

Pre AI Era: Before the widespread adoption of AI, PCG operated within strict parameters and rules. Minecraft leveraged PCG x to unlock endless gameplay potential by crafting diverse, boundless worlds, complete with landscapes, caves, and ecosystems. It intelligently employs gradient noise algorithms like Perlin Noise to maintain coherence amid randomness. 

AI Infusion: AI integration enhanced PCG with player data analysis, adaptive behaviour, and dynamic content generation. In "The Last of Us Part II" AI-driven NPCs show human-like adaptability, using pathfinding algorithms and decision trees to immerse players and keep the gameplay unpredictable, deeply engaging them in the virtual world.

Generative AI Magic: Before Generative AI, graphic content creation relied on rigid rule-based methods. For instance, crafting a forest in Unreal Engine involved configuring surface sampling nodes in its PCG graph and loading pre-built graphical assets for the algorithm to sample. 

Gen AI has the potential to revolutionize this process. Instead of manually configuring individual surfaces, developers can prompt, and the environment is instantly generated. We can adopt a hybrid approach, utilizing PCG graphs for design control while employing Gen AI to create specific 3D elements like trees, bushes, and rocks. Gen AI extends its capabilities beyond environments to include UI elements, cinematics, sound, and automatic subtitles, opening up endless possibilities. The combination of PCG with Gen AI is immensely powerful, and we've only just begun to explore its potential.

While developing a game the most frequent assets we encounter are sprites, textures, skyboxes & 3D meshes Fortunately, Gen AI provides tailored solutions for each. Models like Midjourney and DALL-E swiftly produce compelling sprites and UI assets, ideal for casual game development. Game Artists can save hours by creating the concept art by using simple text prompts & refine it with minor adjustments in software like Photoshop or Substance Sampler, and seamlessly export it for any game engine. 

If you already have a 3D object and want to generate high quality textures inside Unity, you can use Leonardo.ai or PolyHive which can directly generate a PBR (Physically based rendering) texture for your 3D object. PBR textures are essential for photorealistic rendering of 3D objects.

Skyboxes envelop a scene with textures, simulating distant backgrounds such as skies and clouds to enhance immersion and optimize performance. Blockade Labs' Skybox AI accepts a text prompt and crafts a 360-degree skybox environment.

Gen AI can generate highly detailed 3D meshes, saving hours typically spent in Maya3D or Blender. Meshy.AI efficiently generates 3D mesh and textures based on text prompts, ready for any game engine. While Meshy is commercial, frameworks and models like InstantMesh, FlexiCubes, GET3D by NVIDIA, Lumirithmic (for facial scans), and CLIP-Mesh produce top-notch 3D mesh, seamlessly integrated into your graphic workflow.

Performance is vital for a game's success. When using Gen AI to generate content, optimizing assets is crucial. While many LLMs offer customization via techniques like RAG, most graphic asset Gen AI models does not disclose parameter weights or fine-tuning APIs. Thus, optimizing, and personalizing content often involves crafting intelligent prompt templates and refining outputs through intermediary software.

Gen AI is fundamentally changing the game but there are few concerns that have prevented its widespread adoption in art.  To start with is the unresolved question of creative ownership - Who rightfully owns the generated content - the AI, the developer, or both? Then we have a problem of these models of exhibiting inherent bias & producing repetitive and unoriginal content which can significantly dent the reputation of the video games. A prudent approach involves balancing Gen AI-generated content with human intervention for refinement and personalization. While Gen AI holds immense promise, it should complement, not replace, human creativity.

India plays a pivotal role in this revolution. Our greatest asset lies in our diverse talent pool and the relentless spirit of innovation. Not only are we the world's talent powerhouse, but our strides in LLMs and proactive adoption of AI, backed by over 100 start-ups in the Gen AI sector, position us on the cusp of the next big breakthrough. With strategic investment and government initiatives, India has the potential to redefine global standards and spearhead the future of interactive entertainment.

At present, the mainstream adoption of Generative AI in commercial video games is still in its infancy, with most applications being experimental or limited to indie titles. However, this is poised to change quickly, as several game studios plan to employ AI for crafting quests, environments, and entire game worlds in their future releases. Despite the transformative impact of Generative AI on PCG, there are valid concerns. Yet, by employing it judiciously and maintaining transparency, it can be the silver bullet to churn out masterpieces quicker and faster.

Author: Shouvik Mazumdar, Senior Director, Experience Engineering, Ascendion