Unity – Creating a custom map editor (part 2)

Introduction

Hello everyone and welcome back for this second article about custom map editors.

I know this second part has been delayed for a time, the truth is that we are really busy with the making of Radiant Blade. In any case, it’s time to get back to business, so, shall we? 🙂

We left the previous part as follows:

  • Custom UI to select a prefab
  • Placing prefabs on a grid in the editor view

If you missed it, no worry, it will remain available here:

Part 1 of “Creating a custom map editor”

Here is the plan for this part:

  • Adding layers to our map
  • Creating an abstract model of our map for efficiency
  • Removing a placed prefab with ctrl+left click
  • Replacing prefabs instead of stacking them

Bonus: I will need to teach you some important things about how Unity actually works to get you there.

Gearing up

For this article, we are going to use the free tileset available here:

https://www.gameart2d.com/free-platformer-game-tileset.html

Thank you Game Art 2D 🙂

Agreed upon code

In the previous part, the code was mostly used as a tool to explain Unity features and given in small parts. To keep going, we need to make sure that we are using the same base code.
To simply things, here is a revisited code of the previous article:

using UnityEngine;
using UnityEditor;

using System.Collections.Generic;

public class MyMapEditor : EditorWindow
{
    #region Attributes
    // A list containing the available prefabs.
    [SerializeField]
    private List<GameObject> palette = new List<GameObject>();

    [SerializeField]
    private int paletteIndex;

    private string path = "Assets/Editor Default Resources/Palette";
    private Vector2 cellSize = new Vector2(1f, 1f);

    private bool paintMode = false;
    #endregion

    #region UI
    // The window is selected if it already exists, else it's created.
    [MenuItem("Window/My Map Editor")]
    private static void ShowWindow()
    {
        GetWindow(typeof(MyMapEditor));
    }

    void OnFocus()
    {
        SceneView.onSceneGUIDelegate -= this.OnSceneGUI; // Don't add twice
        SceneView.onSceneGUIDelegate += this.OnSceneGUI;

        RefreshPalette(); // Refresh the palette (can be called uselessly, but there is no overhead.)
    }

    // Called to draw the MapEditor windows.
    private void OnGUI()
    {
        paintMode = GUILayout.Toggle(paintMode, "Start painting", "Button", GUILayout.Height(60f));

        // Get a list of previews, one for each of our prefabs
        List<GUIContent> paletteIcons = new List<GUIContent>();
        foreach (GameObject prefab in palette)
        {
            // Get a preview for the prefab
            Texture2D texture = AssetPreview.GetAssetPreview(prefab);
            paletteIcons.Add(new GUIContent(texture));
        }

        // Display the grid
        paletteIndex = GUILayout.SelectionGrid(paletteIndex, paletteIcons.ToArray(), 6);
    }

    // Does the rendering of the map editor in the scene view.
    private void OnSceneGUI(SceneView sceneView)
    {
        if (paintMode)
        {
            Vector2 cellCenter = GetSelectedCell(); // Refactoring, I moved some code in this function

            DisplayVisualHelp(cellCenter);
            HandleSceneViewInputs(cellCenter);

            // Refresh the view
            sceneView.Repaint();
        }
    }

    private void DisplayVisualHelp(Vector2 cellOrigin)
    {
        // Vertices of our square
        Vector3 topLeft = cellOrigin + Vector2.up * cellSize;
        Vector3 topRight = cellOrigin + Vector2.right * cellSize + Vector2.up * cellSize;
        Vector3 bottomLeft = cellOrigin;
        Vector3 bottomRight = cellOrigin + Vector2.right * cellSize;

        // Rendering
        Handles.color = Color.green;
        Vector3[] lines = { topLeft, topRight, topRight, bottomRight, bottomRight, bottomLeft, bottomLeft, topLeft };
        Handles.DrawLines(lines);
    }
    #endregion

    #region MapEdition
    private void HandleSceneViewInputs(Vector2 cellCenter)
    {
        // Filter the left click so that we can't select objects in the scene
        if (Event.current.type == EventType.Layout)
        {
            HandleUtility.AddDefaultControl(0); // Consume the event
        }

        // We have a prefab selected and we are clicking in the scene view with the left button
        if (paletteIndex < palette.Count && Event.current.type == EventType.MouseDown && Event.current.button == 0)
        {
            // Create the prefab instance while keeping the prefab link
            GameObject prefab = palette[paletteIndex];
            GameObject gameObject = PrefabUtility.InstantiatePrefab(prefab) as GameObject;
            gameObject.transform.position = cellCenter + cellSize * 0.5f;

            // Allow the use of Undo (Ctrl+Z, Ctrl+Y).
            Undo.RegisterCreatedObjectUndo(gameObject, "");
        }
    }
    #endregion

    void OnDestroy()
    {
        SceneView.onSceneGUIDelegate -= this.OnSceneGUI;
    }

    private void RefreshPalette()
    {
        palette.Clear();

        System.IO.Directory.CreateDirectory(path);
        string[] prefabFiles = System.IO.Directory.GetFiles(path, "*.prefab");
        foreach (string prefabFile in prefabFiles)
            palette.Add(AssetDatabase.LoadAssetAtPath(prefabFile, typeof(GameObject)) as GameObject);
    }

    Vector2 GetSelectedCell()
    {
        // Get the mouse position in world space such as z = 0
        Ray guiRay = HandleUtility.GUIPointToWorldRay(Event.current.mousePosition);
        Vector3 mousePosition = guiRay.origin - guiRay.direction * (guiRay.origin.z / guiRay.direction.z);

        // Get the corresponding cell on our virtual grid
        Vector2Int cell = new Vector2Int(Mathf.FloorToInt(mousePosition.x / cellSize.x), Mathf.FloorToInt(mousePosition.y / cellSize.y));
        Vector2 cellCenter = cell * cellSize;

        return cellCenter;
    }
}

Three main things changed in this code:

  • If the palette folder doesn’t exist, it is created
  • Cells are now aligned with the scene view grid
  • The cell size has been modified from 2 to 1.

Integrating the graphics

Since we downloaded some assets, it can be good to integrate them properly. You may already know this, but it can’t hurt to recall it.

The grid seen in Unity is a grid such as 1 cell length is 1 unit. It means that adding 1 to the transform position on any axis will move the object of exactly one Unity cell.

A sprite in Unity has a property called “Pixel per unit”. It’s the number of pixels displayed within one Unity cell.

Relation between pixels and Unity scene view

In our case, the tiles textures are 128 pixels wide, so if we input that value in the pixels per unit property, we naturally have our sprites occupying exactly one Unity cell.

With the proper Pixels Per Unity value, our prefabs can be created without any weird scaling trick.

A prefab for our Map Editor, as simple as it can be

And using our custom map editor, we can place them flawlessly in the scene view:

Our platform with three parts aligning flawlessly

A clean job! Except for one small thing: the grass is not hitting the top part of the cell, it’s floating in-between.

This is due to our sprite having a weird height of 93 pixels. The pivot is by default at the center of the sprite, which makes our sprite top point at the middle of the cell, plus half the sprite size: 64+93/2=110.5 pixels of height.

Fixing it is easy, we need to compensate for those missing pixels by lowering the pivot point of 128-110.5=17.5 pixels.

Since the initial pivot is at 93/2=46.5 pixels, it gives us a new pivot with a y coordinate of 46.5-17.5=29 pixels.

Which gives us the following result:

The platform aligned with the scene grid

Notice how the top of the grass is perfectly aligned with the top of cell? This is a clean job, let me tell you that much :p

One last example to make sure we are on the same page:

Our tree sprite placed on the grid

I integrated the tree such as its pivot is half a cell above the base of the sprite, which is 128/2=64 pixels. By doing so, I’m putting it exactly at the center of a Unity cell.

If things are going wrong, you can visualize the pivot using the pivot mode (underlined in red in this screenshot).

Sorting layers

For our assets, I will use three sorting layers:

The tiles, the water and the crates will be on the “Main” layer. The trees will be in the “Background” while the bush will be in the “Foreground”.

Abstract model

So far, we are simply placing objects in the scene view. It’s not actually that impressive, is it?

To move further, we need to modelize our map in a way such as we will be able to apply operations on it.
This is done by maintaining an optimized representation of our current map alongside the actual scene.

Objectives

There are a few main features required from our abstract model

  • Should be able to tell us what occupies a cell
  • Should be able to add a new object in a specific cell
  • Should be able to remove an object from a specific cell
  • Should be able to distinguish layers (a brush can occupy the same cell as a tree)

And of course, it needs to be fast.

In our description, we ended with two concepts: the notion of cell and the notion of layer.

The key to our map

If we take a closer look at what we are doing, we can notice that we are working on a setup assigning objects to keys:

Coordinates throughout our scene grid

A key here is fully described by a set of coordinates and a layer. By having a well defined key, we are going to simplify things a great deal for our model, becau

se key-value associations are something that we can easily manipulate through a Dictionary.

To stick to this notion of key, let’s create an appropriate structure.

using UnityEngine;
using System;

/**
 * Identifier for an object on the map.
 */
public struct MapObjectKey : IEquatable<MapObjectKey>
{
    public int sortingLayerID; // SpriteRenderer.sortingLayerID
    public Vector2Int cell; // Coordinates on the virtual grid

    override public bool Equals(object obj)
    {
        return obj is MapObjectKey && Equals((MapObjectKey) obj);
    }

    public bool Equals(MapObjectKey other)
    {
        return sortingLayerID == other.sortingLayerID && cell == other.cell;
    }

    static public bool operator ==(MapObjectKey object1, MapObjectKey object2)
    {
        return object1.sortingLayerID == object2.sortingLayerID && object1.cell == object2.cell;
    }

    static public bool operator !=(MapObjectKey object1, MapObjectKey object2)
    {
        return object1.sortingLayerID != object2.sortingLayerID || object1.cell != object2.cell;
    }

    public override int GetHashCode()
    {
        return sortingLayerID + cell.GetHashCode();
    }
}

Constraints

The model we are going to use is basically a wrapper around a Dictionary. Simple? Well, actually, not really.

We have several problems due to how Unity works internally. Here are a few things we need to know:

  • Our “game” scripts cannot see Editor scripts
  • References are lost when entering or leaving playmode (if they are not serializable)
  • Our editor windows may or may not be opened
  • The user can change the active scene, unloading a map and loading another map, if it even exists
  • Actively scanning the scene to read the map content is slow, especially when we don’t know when to scan

On the other side, we have useful solutions provided by Unity at our disposal:

  • Editor scripts can see “game” scripts
  • “Game” scripts can contain “editor only” code
  • Components can be cleared during compilation without overhead

Okay, I admit, I’m giving you all this a bit suddenly, but there are reasons, believe me. If we move in naively and fall into every possible pitfall, we will still be there next year.

Implementing the editor-only model

So, let’s recap a bit.

We don’t want to scan the scene from our model to retrieve the map content, that would be both slow and bug prone (what if the users delete a game object manually?).
Because of that, we are going to register the map content actively, which means by having an active component on our map objects.
It won’t be an issue: we can clear this component when compiling, avoiding any overhead at runtime.

Our model needs to be accessible from the game side of the code, because a component will register to it.
To do so, we have two choices. Either we use a static class, or we use a game object and attach the model as a component on it.

I have a problem with the second solution: the user can delete the game object and screw with us, so I prefer to use a static class.

Here is our corresponding script for the model:

#if UNITY_EDITOR
using UnityEngine;
using System.Collections.Generic;

/**
 * This class is on the /Scripts part of our project, but it should actually only exist while in Editor.
 * 
 * Just to make sure we are not doing anything wrong, we wrap is with #if UNITY_EDITOR.
 */
public static class MapEditorModel {

    /**
     * This simple Dictionary is the core of our model.
     */
    private readonly static Dictionary<MapObjectKey, MapObject> map = new Dictionary<MapObjectKey, MapObject>();

    /**
     * Adding a game object via its attached MapObject.
     */
    public static void Register(MapObject mapObject)
    {
        map.Add(GetKey(mapObject), mapObject);
    }

    /**
     * Removing a game object via its attached MapObject.
     */
    public static void Remove(MapObject mapObject)
    {
        map.Remove(GetKey(mapObject));
    }

    /**
     * Removing a game object via its cell and its sorting layer ID.
     */
    public static void Remove(Vector2 position, int sortingLayerID)
    {
        map.Remove(new MapObjectKey()
        {
            cell = GetCell(position),
            sortingLayerID = sortingLayerID
        });
    }

    /**
     * Retrieve a game object via its position and its sorting layer ID.
     * 
     * Will return null if there are none.
     */
    public static MapObject Get(Vector2 position, int sortingLayerID)
    {
        MapObject mapObject = null;
        MapObjectKey key = new MapObjectKey()
        {
            cell = GetCell(position),
            sortingLayerID = sortingLayerID
        };

        map.TryGetValue(key,
                        out mapObject);

        return mapObject;
    }

    /**
     * Creates the corresponding key for a map object.
     */
    private static MapObjectKey GetKey(MapObject mapObject)
    {
        return new MapObjectKey()
        {
            cell = GetCell(mapObject.transform.position),
            sortingLayerID = mapObject.SortingLayerID
        };
    }

    /**
     * This should replace the cellSize defined in the Window Editor script.
     * 
     * Defines the size of a cell for our map. Could be moved in a Scriptable Object.
     */
    public readonly static Vector2 cellSize = new Vector2(1f, 1f);

    /**
     * This should replace the corresponding defined in the Window Editor script.
     * 
     * Returns the coordinates of the pointed cell.
     */
    public static Vector2Int GetCell(Vector3 coords)
    {
        return new Vector2Int(Mathf.FloorToInt(coords.x / cellSize.x), Mathf.FloorToInt(coords.y / cellSize.y));
    }
}
#endif

In summary:

  • It’s a static class that can be reached at any time without worry
  • While the code is in /Scripts, it actually contains editor only code, it won’t be present in our build
  • It wraps a Dictionary with utility methods

Note that will script will not compile yet because of its dependency to MapObject. No worry, the missing script comes next.

Implementing the editor-only component

We have a model, now we simply need to use it.

We are going to take advantage of the IProcessSceneWithReport interface:

https://docs.unity3d.com/ScriptReference/Build.IProcessSceneWithReport.html

Basically, we just need to implement the interface on a plain class, Unity will take care of the rest with heavy reflection work at compile-time. For the curious ones:

https://github.com/Unity-Technologies/UnityCsReference/blob/053d3ce1c53ee3dd4ff1aa08522c1284453f0796/Editor/Mono/BuildPipeline/BuildPipelineInterfaces.cs

Apart from this small trick to clean our build, the component itself is nothing scary:

#if UNITY_EDITOR
using UnityEngine;

using UnityEditor.Build;
using UnityEditor.Build.Reporting;

/**
 * Put this component on the Map Editor prefabs.
 */
[RequireComponent(typeof(SpriteRenderer))]
[ExecuteInEditMode]
public class MapObject : MonoBehaviour
{
    [Tooltip("Fill this if the SpriteRenderer is in a child, leave empty otherwise.")]
    [SerializeField] private SpriteRenderer spriteRenderer;

    // Sorting layer of our SpriteRenderer, can be called why we are a prefab, with no prior initialization
    public int SortingLayerID {
        get {
            return (spriteRenderer ? spriteRenderer : GetComponent<SpriteRenderer>()).sortingLayerID;
        }
    }

    /**
     * Called when placing the order in the scene, after entering playmode and after leaving playmode.
     */
    private void Start()
    {
        MapEditorModel.Register(this);
    }


    /**
     * Counterpart of Start, called when removing from the scene, before entering playmode and before leaving playmode.
     */
    private void OnDestroy()
    {
        MapEditorModel.Remove(this);
    }
}


/**
 * Called when compiling, will remove all MapObject components from our scenes.
 */
public class MapObjectProcess : IProcessSceneWithReport
{
    public int callbackOrder { get { return 0; } }

    public void OnProcessScene(UnityEngine.SceneManagement.Scene scene, BuildReport report)
    {
        Debug.Log("MapObjectProcess.OnProcessScene " + scene.name);

        // Get all root objects in the scene.
        foreach (GameObject go in scene.GetRootGameObjects())
        {
            // Get all MapObject components for each root object
            foreach (MapObject mapObject in go.GetComponentsInChildren<MapObject>())
                GameObject.DestroyImmediate(mapObject); // Remove it
        }
    }
}
#endif

One important thing. This component will be present on our prefabs, and our Map Editor will handle those prefabs.
The trick is, prefabs don’t have their Start/Awake methods called, since they are not yet on the scene. Because of this, we end up implementing the SortingLayerID property without relying on the Start method.

Then we only need to update our prefabs by adding our new component to them:

Addid our MapObject component to our prefabs

Updating the map editor

This part is left as an exercise for you. Nothing fancy, really. Our map editor currently maintains a list of game objects:

To maintain some homogeneity into out code, we should convert it into a list of MapObject instead and modify the code accordingly:

It’s no big deal, but since we are going to feed MapObject components to our model, it would be quite boring to resort on GetComponent<MapObject> whenever we need to do something.

Replacing existing tiles

We did a lot of preliminary work here, but now we are ready to move faster. Let’s add an actual feature to our map editor: removing an already existing tile when placing another one on top of it.

This is the code portion that we will improve:

This first version was naive, it simply adds a new prefab in the scene without much sensitivity.

Let’s rewrite it, in a new method of our, to do as follows:

  • Check if a map object already exists at the position
  • Destroy it if it exists
  • Place the new map object

Quite simple, honestly, let’s write it down:

    #region MapEdition
    private void HandleSceneViewInputs(Vector2 cellCenter)
    {
        // Filter the left click so that we can't select objects in the scene
        if (Event.current.type == EventType.Layout)
        {
            HandleUtility.AddDefaultControl(0); // Consume the event
        }

        // We have a prefab selected and we are clicking in the scene view with the left button
        if (paletteIndex < palette.Count && Event.current.type == EventType.MouseDown && Event.current.button == 0)
        {
            // Place a new prefab instance on the map
            MapObject prefab = palette[paletteIndex];
            AddMapObject(cellCenter, prefab);
        }
    }

    /**
     * Remove any already existing object
     */
    private void RemoveMapObject(Vector2 cellCenter, int sortingLayerID)
    {
        MapObject previousObject = MapEditorModel.Get(cellCenter, sortingLayerID);
        if (previousObject)
            Undo.DestroyObjectImmediate(previousObject.gameObject); // Using Undo to allow undo/redo
    }

    /**
     * Add a new object on the map with replacement if anything is already there.
     */
    private void AddMapObject(Vector2 cellCenter, MapObject prefab)
    {
        // Remove any already existing object
        RemoveMapObject(cellCenter, prefab.SortingLayerID);

        MapObject mapObject = PrefabUtility.InstantiatePrefab(prefab) as MapObject;
        mapObject.transform.position = cellCenter + cellSize * 0.5f;

        // Using Undo to allow undo/redo
        Undo.RegisterCreatedObjectUndo(mapObject, "");
    }
    #endregion

I’ve split the method into two parts: the removal and the addition. That way, implementing our next feature, the deletion, will be trivial.

Let’s take a lot at our new tool:

Using the Map Editor with the replacement feature

Removing a tile

We are just missing one feature to get something a little bit more workable with: removing a specific object on demand.

Since we already have to method for this, we just need to call it. In my version, I’ve chosen to use ctrl+click for the deletion, which can be expressed as easily as:

            if (Event.current.control)
                RemoveMapObject(cellCenter, prefab.SortingLayerID);
            else
                AddMapObject(cellCenter, prefab);

“[When clicking…] if we are not pressing control, then remove the object, else add a new one.”

Yup, that’s all.

Conclusion

The map editor is getting somewhere!

One of its strengths is how it handles prefabs. You can decorate your map with creatures, items, and much more at ease, it’s really not just constrained at designing the map itself.

Of course we can go further, yet again. We could have random selection among a set of possibilities when putting a mushroom or a floor, we could have brushes, we could have an isometric grid instead of a classic one.

But all that is just about coding it, and writing about it wouldn’t teach you much, so I will leave it to you.

Don’t hesitate to follow us on Twitter to get news about the upcoming tutorials!

Have a good day, and don’t hesitate to leave a comment 🙂