Tasking the environment

Each episode of the environment is parameterized by the task of the agent. Each task is represented by string containing the converation between the architect and the builder and also by the target structure encoded into 3d voxel grid. Initially, the environment loads the set of tasks from the dataset collected in [1]. Each env.reset samples a new task from the current task set and makes it active. We provide several types of sets of tasks to use with the environment. You can control the set task using the corresponding class, for example RandomTasks, TaskSet, or CustomTasks.

Random tasks

As a bootstrap for learning, we provide the type of tasks which are generated randomly. For this kind of tasks set there will be no conversations, so each goal will have an empty string inside its "chat" observation. You can control the difficulty of generated tasks by changing parameters of RandomTasks class:

class iglu.tasks.task_set.RandomTasks(max_blocks=4, height_levels=1, allow_float=False, max_dist=2, num_colors=1, max_cache=0)

TaskSet that consists of number of randomly generated tasks

Parameters

  • max_blocks (int) – The maximal number of blocks in each task. Defaults to 3.

  • height_levels (int) – How many height levels random blocks can occupy. Defaults to 1.

  • allow_float (bool) – Whether to allow blocks to have the air below. Defaults to False.

  • max_dist (int) – Maximum (Chebyshev) distance between two blocks. Defaults to 2.

  • num_colors (int) – Maximum number of unique colors. Defaults to 1.

  • max_cache (int) – If zero, each .sample_task will generate new task. Otherwise, the number of random tasks to cache. Defaults to 0.

To use random tasks with the environment, you should call .update_taskset method.

import gym
from iglu.tasks import RandomTasks

env = gym.make('IGLUSilentBuilder-v0')
env.update_taskset(RandomTasks(max_blocks=3, max_dist=5, num_colors=3))

for episode in range(10):
    # for each reset call there will be a completely new target
    obs = env.reset()
    done = False

    while not done:
        action = env.action_space.sample()
        obs, reward, done, info = env.step(action)

Custom tasks

CustomTasks class provides an ability to load and use an arbitrary set of tasks. Each task should contain a conversation betwen the architect and the builder; 3d numpy array with target coordinates. There’s no need to specify intermediate steps of building the structure, only the final structure is required.

An example of building simple custom goal with an imagined conversation:

import gym
import numpy as np
from iglu.tasks import CustomTasks

env = gym.make('IGLUSilentBuilder-v0')
custom_grid = np.zeros((9, 11, 11)) # (y, x, z)
custom_grid[:3, 5, 5] = 1 # blue color
env.update_taskset(CustomTasks([
    ('<Architect> Please, build a stack of three blue blocks somewhere.\n'
     '<Builder> Sure.',
     custom_grid)
]))

Here is the correspondense between colors and ids:

from iglu.const import id2color; print(id2color)

{0: 'air', 1: 'blue', 2: 'yellow', 3: 'green', 4: 'orange', 5: 'purple', 6: 'red'}

Minecraft dialogue dataset

All tasks are sampled from the dataset collected in [1]. Each task has its unique id of the format C<id> where <id> is a number. On creation, the env loads all tasks from the dataset.

To load new set of tasks into the environment

import gym
from iglu.tasks import TaskSet

env = gym.make('IGLUSilentBuilder-v0')
env.update_taskset(TaskSet(preset=['C1', 'C2', 'C3']))

By default the env loads task with id C8. To use all available tasks within the set, use TaskSet.ALL list as a preset.

Here is the full list of available goals:

from iglu.tasks.task_set import TaskSet
print('\n'.join(f'{k}: {v}' for k, v in TaskSet.ALL.items()))

C1: bell (29 blocks, 5 colors)
C2: black-hole (20 blocks, 2 colors)
C3: blue-original-L (3 blocks, 1 colors)
C4: flower_new (17 blocks, 4 colors)
C5: overlapping-chain-links (20 blocks, 2 colors)
C6: rectangle-chain (40 blocks, 4 colors)
C7: scissors (23 blocks, 2 colors)
C8: table2 (12 blocks, 2 colors)
C9: asterisk (33 blocks, 2 colors)
C10: concentric_semicircles (21 blocks, 3 colors)
C11: broken_heart (21 blocks, 2 colors)
C12: diagonal-Ls (18 blocks, 6 colors)
C13: eye (29 blocks, 3 colors)
C14: diagonal-zigzag (16 blocks, 3 colors)
C15: double_stairs (36 blocks, 4 colors)
C16: bloody-sword (24 blocks, 4 colors)
C17: orange-flat-original-L (3 blocks, 1 colors)
C18: overlapping-reticles (12 blocks, 3 colors)
C19: slide_with_arch (48 blocks, 3 colors)
C20: rainbow-lasso (9 blocks, 6 colors)
C21: spectacles (29 blocks, 2 colors)
C22: smiley (6 blocks, 5 colors)
C23: suspension_bridge (68 blocks, 4 colors)
C24: cup (46 blocks, 4 colors)
C25: music-notes (18 blocks, 2 colors)
C26: rainbow-diagonal-stack (15 blocks, 6 colors)
C27: funny-Ts (25 blocks, 4 colors)
C28: duck (21 blocks, 2 colors)
C29: tetris (20 blocks, 5 colors)
C30: 3d-triangle (20 blocks, 6 colors)
C31: hot (20 blocks, 3 colors)
C32: bigger-original-L (5 blocks, 1 colors)
C33: connected_diamonds_with_missing_blocks (14 blocks, 2 colors)
C34: complex_five (13 blocks, 3 colors)
C35: strange-triangles (20 blocks, 6 colors)
C36: street_lamps (31 blocks, 3 colors)
C37: embedded_fives (56 blocks, 6 colors)
C38: l-shape
C39: three-squares (24 blocks, 3 colors)
C40: cross-stack (9 blocks, 6 colors)
C41: multiple_concepts (19 blocks, 3 colors)
C42: rubiks (18 blocks, 4 colors)
C43: girl_with_frock (29 blocks, 3 colors)
C44: hollow_diseased_rectangle (19 blocks, 4 colors)
C45: u-3 (14 blocks, 2 colors)
C46: wtf (11 blocks, 6 colors)
C47: staircase (18 blocks, 3 colors)
C48: umbrella (37 blocks, 4 colors)
C49: letter-cube (36 blocks, 4 colors)
C50: 3d-x_s (15 blocks, 3 colors)
C51: interlocking-magnets (16 blocks, 4 colors)
C52: hut_and_trees (31 blocks, 3 colors)
C53: funny-cube (20 blocks, 4 colors)
C54: 3d-diagonals (10 blocks, 3 colors)
C55: chains (19 blocks, 3 colors)
C56: color-wheel (25 blocks, 4 colors)
C57: four-three (17 blocks, 2 colors)
C58: ducks_in_pool (24 blocks, 2 colors)
C59: parallel-bars (20 blocks, 4 colors)
C60: diagonal-2 (20 blocks, 6 colors)
C61: chair2 (19 blocks, 2 colors)
C62: chair3 (32 blocks, 3 colors)
C63: question (6 blocks, 1 colors)
C64: torch (17 blocks, 2 colors)
C65: superhero (15 blocks, 3 colors)
C66: connected_triangles (30 blocks, 3 colors)
C67: storm-cloud (25 blocks, 2 colors)
C68: christmas-tree (25 blocks, 6 colors)
C69: pickup_truck (38 blocks, 5 colors)
C70: random_snake (29 blocks, 2 colors)
C71: connected_rows_with_missing_blocks (15 blocks, 4 colors)
C72: cat (37 blocks, 4 colors)
C73: x_equals_y (18 blocks, 3 colors)
C74: random_objects (14 blocks, 4 colors)
C75: creative-five (11 blocks, 6 colors)
C76: double_tripod (16 blocks, 5 colors)
C77: 3d-maze (40 blocks, 5 colors)
C78: random_objects_2 (24 blocks, 3 colors)
C79: connected_crosses (25 blocks, 4 colors)
C80: lunging-man (18 blocks, 4 colors)
C81: cube-x (17 blocks, 3 colors)
C82: eiffel-tower (40 blocks, 4 colors)
C83: pyramid-table (33 blocks, 4 colors)
C84: trident (13 blocks, 2 colors)
C85: upvote (9 blocks, 2 colors)
C86: butterfly (24 blocks, 3 colors)
C87: sickles (10 blocks, 1 colors)
C88: sun_and_birds (19 blocks, 2 colors)
C89: m-and-m (23 blocks, 2 colors)
C90: wind-farm (30 blocks, 4 colors)
C91: castle (41 blocks, 5 colors)
C92: cherries (16 blocks, 2 colors)
C93: L-back-to-back (8 blocks, 2 colors)
C94: disappearing-tunnel (35 blocks, 6 colors)
C95: chimney (26 blocks, 3 colors)
C96: four-corners (16 blocks, 4 colors)
C97: omega-u-corner (14 blocks, 2 colors)
C98: insects (24 blocks, 6 colors)
C99: keys (35 blocks, 2 colors)
C100: wannabe_olympic_rings (42 blocks, 4 colors)
C101: connected_tunnels (31 blocks, 3 colors)
C102: man_on_broom (16 blocks, 4 colors)
C103: rotating_chair (19 blocks, 1 colors)
C104: clothesline (36 blocks, 2 colors)
C105: lol_ethylene (34 blocks, 3 colors)
C106: minimal_tree (18 blocks, 2 colors)
C107: colorful_alien (20 blocks, 3 colors)
C108: illinois-i (29 blocks, 2 colors)
C109: waving-flag (11 blocks, 2 colors)
C110: color-loop (16 blocks, 4 colors)
C111: color-cube (16 blocks, 4 colors)
C112: color-spiral (33 blocks, 5 colors)
C113: numbers (43 blocks, 4 colors)
C114: e3 (18 blocks, 2 colors)
C115: hollow_stools (23 blocks, 4 colors)
C116: diagonal_arch_and_triangle (17 blocks, 4 colors)
C117: pyramid_with_horns (20 blocks, 4 colors)
C118: random_objects_3 (12 blocks, 2 colors)
C119: funny_chair (38 blocks, 3 colors)
C120: h_1_plus (14 blocks, 2 colors)
C121: 2d_giraffe (19 blocks, 3 colors)
C122: funny_bird (25 blocks, 4 colors)
C123: footballer (26 blocks, 3 colors)
C124: connected_triangles_2 (29 blocks, 3 colors)
C125: fail_hourglass (22 blocks, 3 colors)
C126: teapot (20 blocks, 2 colors)
C127: backpack (26 blocks, 3 colors)
C128: weird_chain (16 blocks, 4 colors)
C129: rhombuses_with_ring (34 blocks, 3 colors)
C130: differently_oriented_squares (24 blocks, 2 colors)
C131: table_with_objects (35 blocks, 6 colors)
C132: diagonal_slab_with_pattern (20 blocks, 2 colors)
C133: adjacent_gates (28 blocks, 3 colors)
C134: parachute (28 blocks, 6 colors)
C135: bent-arrow (11 blocks, 1 colors)
C136: waving-man (17 blocks, 4 colors)
C137: no-u-turn (39 blocks, 2 colors)
C138: color-Us (42 blocks, 6 colors)
C139: minimalistic-mouse (16 blocks, 3 colors)
C140: weird-rows (28 blocks, 4 colors)
C141: dancing-banana (19 blocks, 4 colors)
C142: potted-weeds (28 blocks, 3 colors)
C143: drooping-flower (21 blocks, 5 colors)
C144: curly-R (20 blocks, 3 colors)
C145: pulse (19 blocks, 1 colors)
C146: boxy-to-diagonal (17 blocks, 5 colors)
C147: pi (26 blocks, 4 colors)
C148: headphones (40 blocks, 4 colors)
C149: drink-with-umbrella (37 blocks, 4 colors)
C150: checkerboard (16 blocks, 6 colors)
C151: hanger (16 blocks, 3 colors)
C152: ballerina (24 blocks, 3 colors)
C153: cursive-ok (27 blocks, 2 colors)
C154: gateway-to-stairs (52 blocks, 6 colors)
C155: sickles-difficult (24 blocks, 2 colors)
C156: sun_and_birds-difficult (19 blocks, 2 colors)
C157: butterfly-difficult (24 blocks, 3 colors)

References

1(1,2)

Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 5405–5415. Florence, Italy, July 2019. Association for Computational Linguistics. URL: https://www.aclweb.org/anthology/P19-1537.