# Deepdrive Sim Unreal based simulator and Python interface to Deepdrive ## Usage Checkout our [main repo](https://github.com/deepdrive/deepdrive) ## Simulator docs [Simulator docs](https://simdocs.deepdrive.io/docs) for making changes to the sim. # Deepdrive Sim Docs ## Setup [Development setup instructions](/docs/setup) ## Requirements * Linux * Python 3.6+ * 20GB disk space * 8GB RAM NOTE: Windows support has been removed due to a mismatch of resources and interest. The main impediment to Windows working is the lack of an embedded python interpreter and issues installing the correct packages from uepy-requirements.txt. ## Development Tips ### VERSION file This is used for checking compatibility between sim and agents and will update automatically after building in Unreal. Please check it in. Also if you make a backwards incompatible change, i.e. to the shared memory interface, bump the minor version in the [MAJOR\_MINOR\_VERSION](https://github.com/deepdrive/deepdrive-sim/tree/2ee6713413b7b8f5e1fbeef0cc067b04ac2a2dc7/docs/Content/Data/MAJOR_MINOR_VERSION/README.md) file. ### Clean builds You'll often want to run `clean.sh` or `clean.bat` after pulling in changes, especially to the plugin as Unreal will spuriously cache old binaries. ### PyCharm If you open an Unreal project in Pycharm, add Binaries, Build, Content, Intermediate, and Saved to your project’s “Excluded” directories in Project Structure or simply by right clicking and choosing “Mark Directory as” => “Excluded”. Keeping these large binary directories in the project will cause PyCharm to index them. Do the same with these directories (Binaries, Build, Content, Intermediate, and Saved) within any of the Plugins in the Plugins folder. Windows - Package via the Unreal Editor ### Setting key binds Unreal->Project Settings->Input->Action Mappings OR in Blueprints->Find (uncheck Find in Current Blueprint Only) and search for the input key, i.e. J. # Setup ## Add yout GitHub account to Epic's GitHub organization First off, [associate your GitHub username with your Unreal account](https://www.unrealengine.com/en-US/ue4-on-github) to get access to Unreal sources on GitHub. ## Set your python bin This is where the Python extension is installed which enables communication between Unreal and Python. ### Option 1: Automatically set the python bin Setup the [deepdrive](https://github.com/deepdrive/deepdrive) project which creates your `~/.deepdrive/python_bin` ### Option 2: Manually set the python bin Create `~/.deepdrive/python_bin` on Unix or `%HOMEPATH%\.deepdrive\python_bin` on Windows and point it to the Python executable you want the deepdrive Python extension to be installed to, i.e. ``` /home/[YOU]/.local/share/virtualenvs/my-env/bin/python ``` ## OS specific setup To finish your development setup, follow one of these guides * [Linux](/docs/setup/linux) * [Windows](/docs/setup/windows) # Linux Development We support the following local development setups 1. [**Export from prebuilt docker image**](/docs/setup/linux/ue4docker-export) (recommended), i.e. ue4-docker installed build - Builds in docker, then installs to your local system. 2. [**Run everything inside docker**](/docs/setup/linux/run-in-docker) (promising, but not heavily used) - Uses X11 forwarding view Unreal running in docker 3. [**Traditional**](/docs/setup/linux/traditional-install) - Run Unreal per the standard Epic instructions (Use this if you want to use an existing Unreal 4.21 installation) # Running the Unreal Editor from a Docker container ## Contents * [Prerequisites](/docs/setup/linux/run-in-docker#prerequisites) * [General instructions](/docs/setup/linux/run-in-docker#general-instructions) * [Step 1: Obtaining the Docker image](/docs/setup/linux/run-in-docker#step-1-obtaining-the-docker-image) * [Step 2: Starting the container](/docs/setup/linux/run-in-docker#step-2-starting-the-container) * [Step 3: Building the project](/docs/setup/linux/run-in-docker#step-3-building-the-project) * [Step 4: Running the Editor](/docs/setup/linux/run-in-docker#step-4-running-the-editor) * [Step 5: (Optional) Running other commands inside the container](/docs/setup/linux/run-in-docker#step-5-optional-running-other-commands-inside-the-container) * [Step 6: Stopping the container](/docs/setup/linux/run-in-docker#step-6-stopping-the-container) * [Deepdrive-specific instructions](/docs/setup/linux/run-in-docker#deepdrive-specific-instructions) * [Additional prerequisites](/docs/setup/linux/run-in-docker#additional-prerequisites) * [Obtaining the Deepdrive Docker image](/docs/setup/linux/run-in-docker#obtaining-the-deepdrive-docker-image) * [Additional arguments for Step 2](/docs/setup/linux/run-in-docker#additional-arguments-for-step-2) ## Prerequisites In order to run the Unreal Editor from within a Docker container, you will need the following: * A Linux distribution with X11 (and optionally PulseAudio if audio output is required) * The official NVIDIA GPU drivers (open source drivers such as Nouveau will not work) * [Docker](https://www.docker.com/) * The [NVIDIA Docker runtime](https://github.com/NVIDIA/nvidia-docker) ## General instructions ### Step 1: Obtaining the Docker image Before you can start working with an Unreal Editor container, you will first need to build or pull a suitable container image. Voyage employees will be working with the Deepdrive Docker image, which is documented in the section [Obtaining the Deepdrive Docker image](/docs/setup/linux/run-in-docker#obtaining-the-deepdrive-docker-image). ### Step 2: Starting the container To start a container from which the Unreal Editor can be run, open a terminal window in the directory containing your Unreal project's source code and execute the following command (**$IMAGE\_NAME** should be replaced with the name of the Docker image that you obtained in Step 1): ```bash docker run --rm -ti --runtime=nvidia --name=unreal_editor "-v$HOME/.config/Epic:/home/ue4/.config/Epic" -v`pwd`:/projectdir -w /projectdir -v/tmp/.X11-unix:/tmp/.X11-unix:rw -e DISPLAY -v/run/user/$UID/pulse:/run/user/1000/pulse $IMAGE_NAME bash ``` The meaning of each of the arguments to the `docker run` command is as follows: * `--rm` ensures the container will be deleted after it has finished running. This ensures you have a clean environment to work from each time you start the image. It is important to note that no data will be lost when the container is deleted, since the project source directory and the Unreal Editor's configuration directory are both bind-mounted from the host system's filesystem. * `-ti` runs the container in interactive mode, which allows you to run commands from the bash shell inside the container in much the same manner as you would on the host system. * `--runtime=nvidia` instructs Docker to use the NVIDIA Docker runtime instead of the default Docker runtime. This allows the container to access the GPU resources of the host system. * `--name=unreal_editor` assigns the identifier "unreal\_editor" to the container. This is useful if you need to run other Docker commands that manipulate or interact with the container while it is running. * `-v$HOME/.config/Epic:/home/ue4/.config/Epic` bind-mounts the host filesystem directory `~/.config/Epic` into the container. This directory is where the Unreal Editor stores its configuration data and compiled shader cache. Storing this data on the host filesystem allows the Unreal Editor to remember information from previous runs and ensures it behaves in a manner more consistent with running the Editor directly on the host system. * ``-v`pwd`:/projectdir`` bind-mounts the current working directory into the container. This allows the Unreal Editor to access your Unreal project's source code. * `-w /projectdir` sets the default working directory of the container to the directory containing your Unreal project's source code. * `-v/tmp/.X11-unix:/tmp/.X11-unix:rw` bind-mounts the host system's X11 socket into the container. This allows the Unreal Editor to create graphical windows that will be displayed on the host system. * `-e DISPLAY` propagates the value of the `DISPLAY` environment variable from the host system to the container. This is needed for correct X11 display behaviour. * `-v/run/user/$UID/pulse:/run/user/1000/pulse` bind-mounts the host system's PulseAudio socket into the container. This is only necessary if audio output is required. * `$IMAGE_NAME` is a placeholder that should be replaced with the name of the Docker image that you obtained in Step 1. * `bash` starts an interactive bash shell within the container. ### Step 3: Building the project Unless you are working with a Blueprint-only Unreal project, you will need to compile your project's C++ source code before you can run the Unreal Editor. You can do this by running the following command inside the container's interactive bash shell: ```bash ue4 build ``` ### Step 4: Running the Editor Once the source code for your Unreal project has been compiled, you can start the Unreal Editor by running the following command inside the container's interactive bash shell: ```bash ue4 run ``` If everything is working correctly, you should see the Unreal Editor splash screen appear while the Editor is loading, followed by the window for the Editor itself. ### Step 5: (Optional) Running other commands inside the container If you would like to run additional commands inside the container while the Unreal Editor is running, you can attach additional interactive bash shells to the container by running the following command: ``` docker exec -ti unreal_editor bash ``` This command can be run multiple times to attach multiple interactive shells. It is important to note that all additional interactive shells will be terminated when the container is stopped, which happens the moment that the original bash shell (the one created when the container was started) exits. Closing any of the additional attached shells will not stop the container. ### Step 6: Stopping the container Once you have closed the Unreal Editor and returned to the container's interactive bash shell, you can stop the container by simply closing the bash shell: ```bash exit ``` Note that this will also terminate any additional interactive shells that you may have attached to the container while it was running. ## Deepdrive-specific instructions ### Additional prerequisites Because the Docker image for Deepdrive contains CUDA 10.0, the version number of the NVIDIA GPU driver matters. You will need to ensure you have a [version of the NVIDIA GPU driver compatible with CUDA 10](https://github.com/NVIDIA/nvidia-docker/wiki/CUDA#requirements). ### Obtaining the Deepdrive Docker image Before you can pull the Docker image for Deepdrive, you will need to login to Docker Hub: ```bash # This will prompt for your Docker ID and password docker login ``` Once you have authenticated, the Docker image for Deepdrive can then be pulled by executing the following command: ```bash docker pull deepdriveio/ue4-deepdrive-deps:latest ``` If you encounter permission issues when pulling the image then you may need to contact Craig Quiter to have your Docker Hub account added to the [deepdriveio organisation](https://hub.docker.com/u/deepdriveio/). ### Additional arguments for Step 2 If you intend to run the Deepdrive deep learning agents inside the container, you will need to bind-mount the directory containing the [Deepdrive source code](https://github.com/deepdrive/deepdrive) in addition to the source code for the Deepdrive simulator Unreal project: ```bash # These are the original arguments specified in Step 2: -v`pwd`:/projectdir -w /projectdir # Replace them with these arguments: -v`pwd`:/simdir -w /simdir -v/path/to/deepdrive:/deepdrivedir ``` Replace `/path/to/deepdrive` with the appropriate host filesystem path for the directory containing the Deepdrive source code. # Export from prebuilt docker image ## Prerequisites * NVIDIA Docker 2 Make sure the ue4-ci-helpers package is up to date ``` pip install --upgrade ue4-ci-helpers pip install ue4-docker ``` ## Build the ue4-full image ## Build the deepdriveio/ue4-deepdrive-deps image ## Export Unreal to your local filesystem ``` ue4-docker export installed deepdriveio/ue4-deepdrive-deps:latest ~/UnrealInstalled ``` Set the default version of Unreal with the [ue4cli](https://pypi.org/project/ue4cli/) ``` ue4 setroot ~/UnrealInstalled ``` ## Build and run the project against the exported UnrealEngine Now you're ready to build and run the project. This is what you'll do any time you've pulled in the latest changes from GitHub as well. Download maps ``` python Packaging/package.py --download-jamestown ``` ### Clean and build Do the following the first time you export or anytime you pull latest from deepdrive-sim. #### Clean your deepdrive-sim project directory to clear any stale build data ``` ./clean_all.sh ``` #### Build and run - takes \~5 minutes ``` ue4 build ue4 run ``` # Traditional Unreal Setup * Clone this repo * Clone our [UnrealEnginePython fork](https://github.com/deepdrive/UnrealEnginePython) into the root of this project (not a submodule as we are going to make this a binary only) ``` cd Plugins git clone https://github.com/deepdrive/UnrealEnginePython ``` * Tip: To avoid rebuilding UnrealEnginePython, move `Plugins/UnrealEnginePython` to your Engine plugins folder, i.e. `/Engine/Plugins/UnrealEnginePython/` after you've built the deepdrive-sim project with the plugin in `deepdrive-sim/Plugins`. ``` git clone git@github.com:EpicGames/UnrealEngine --branch 4.21 # or if you are using http: # git clone https://github.com/EpicGames/UnrealEngine --branch 4.21 ``` `` is now populated with the Unreal Engine source. ## Build Unreal *Takes an hour with 12 cores* ``` cd UnrealEngine ./Setup.sh && ./GenerateProjectFiles.sh && make ``` More details on building Unreal [here](https://wiki.unrealengine.com/Building_On_Linux), though the above commands should be sufficient. ## Get the substance plugin Download the Substance plugin from [here](https://forum.allegorithmic.com/index.php/topic,26732.0.html) and unzip the `Plugins/Runtime/Substance` folder within it to `/Plugins/Runtime` for Unreal 4.21. For other releases, see [here](https://forum.allegorithmic.com/index.php/board,23.0.html) or you can just use the sources downloaded by Windows / Mac marketplace. ## Run the editor ``` ./Engine/Binaries/Linux/UE4Editor ``` Finally open the deepdrive-sim uproject file within the editor. # Windows Development * Use Visual Studio 2015 with the [C++ build tools](https://stackoverflow.com/a/31955339) * Get Unreal v4.21 via the Epic Launcher -> Unreal Engine tab -> Library * Optionally download Unreal sources and debugging symbols in the Epic Launcher ![Windows Unreal Install Options](https://i.imgur.com/Khxc6HV.jpg) * Download UnrealEnginePython \[python binaries] into `Plugins/UnrealEnginePython/Binaries` * Install the Substance Plugin through the Marketplace in Epic Launcher * Open DeepDrive.uproject with the version of Unreal Editor you just installed - if there are errors, check `Saved/Logs` for details * Refresh / Create Visual Studio project * Open the Visual Studio Project * Close Unreal * Right click the DeepDrive project and set as Startup Project, debug... * To run game full speed when the window is not focused * Uncheck Edit->Editor Preferences->Use Less CPU when in Background # Compiling changes to plugins If you make a change to the DeepDrivePlugin, you'll often have to recompile the module to see the changes. To do so in Unreal Editor, open Window->Developer Tools->Modules and search for DeepDrivePlugin. Then hit Recompile to compile your plugin changes. # Vulkan With Unreal 4.21, Vulkan is automatically supported. I've tried it with NVIDIA 384 drivers and things crash, but newer drivers may work. If you experience crashes and see mentions to Vulkan in the logs, you can ensure OpenGL is used by uinstalling these debian packages ``` sudo apt remove libvulkan1 mesa-vulkan-drivers vulkan-utils ``` # python\_bindings ## Building the python bindings ``` install-dev-bindings.sh ``` ## Pushing bindings to PyPi Pushing to the release branch causes CI to publish wheels to the PyPi cheese shop. ``` git push origin master && git push origin master:release ``` # Unreal Logging Command line example ``` -LogCmds="global Verbose, LogPython Verbose, LogAnimMontage off, LogDeepDriveAgent VeryVerbose" ``` In DefaultEngine.ini or Engine.ini: ``` [Core.Log] global=[default verbosity for things not listed later] [cat]=[level] foo=verbose break ``` ## Our log categories ``` Defined with DEFINE_LOG_CATEGORY macro ``` ``` LogSunSimulationComponentLogSunLightSimulator LogDeepDriveAgentControllerBase LogDeepDriveAgentSteeringController LogDeepDriveAgent LogDeepDriveSimulationServer LogDeepDriveConnectionThread LogDeepDriveAgentLocalAIController LogDeepDriveAgentSpeedController LogDeepDriveSimulationCaptureProxy LogDeepDriveSimulationMessageHandler LogDeepDriveSimulationServerProxy LogDeepDriveSimulation LogDeepDriveSplineTrack LogSharedMemCaptureMessageBuilder LogSharedMemCaptureSinkWorkerLogSharedMemCaptureSinkWorker LogSharedMemCaptureSinkComponent DeepDriveCaptureProxy DeepDriveCaptureComponent LogDeepDriveCapture LogCaptureBufferPool LogDeepDriveConnectionListener LogDeepDriveClientConnection LogDeepDriveServer LogDeepDrivePlugin LogSharedMemoryImpl_Linux LogPython ``` ## Verbosities Fatal ``` Fatal level logs are always printed to console and log files and crashes even if logging is disabled. ``` Error ``` Error level logs are printed to console and log files. These appear red by default. ``` Warning ``` Warning level logs are printed to console and log files. These appear yellow by default. ``` Display ``` Display level logs are printed to console and log files. ``` Log ``` Log level logs are printed to log files but not to the in-game console. They can still be viewed in editor as they appear via the Output Log window. ``` Verbose ``` Verbose level logs are printed to log files but not the in-game console. This is usually used for detailed logging and debugging. ``` VeryVerbose ``` VeryVerbose level logs are printed to log files but not the in-game console. This is usually used for very detailed logging that would otherwise spam output. ``` # Packaging ## Setup ``` pip install -r Packaging/requirements.txt ``` ## Linux To package a new binary version in Linux run ``` Packaging/package-linux.sh ``` NOTE: Uploads to S3 are done automatically through CI after tests have passed. To upload to S3 anyway: ``` python Packaging/package.py --upload-only ``` ## Windows In Windows we go through the Unreal Editor packaging interface, and manually add UnrealEnginePython. TODO: Fill this out. # Architecture deepdrive-sim consists of two main components. 1. ] The 'game' or simulation, including all the assets, materials, blueprints, lighting, etc... a la a traditional Unreal game project 2. ] The liaison between Unreal and your Python, delivering sensor data from the game and receiving control and configuration commands through a combined C++ Python extension / Unreal Plugin. ## DeepDrivePlugin technical details ### Abstract The DeepDrivePlugin allows extracting images and depth from several cameras via shared memory along with controlling the car and configuring the simulation over TCP sockets. ### Overview The functionality of the DeepDrivePlugin can divided into 2 main domains: * Capturing * Control Capturing provides the means to extract an arbitrary number of live streams out of the simulated world including current state data of the agent. Control provides the means to control capturing. ![Overview](/files/-Lzt9l0RhLQu0cBji3BJ) ### Capturing Capturing is the process of regularly taking a snapshot of the simulated world and processing the data (i.e. transferring snapshot data to a client). Capturing is separated into these 2 steps: 1. Taking the snapshot 2. Processing the data Currently a snapshot consists of the following data: * state data of agent * upto n camera streams whereas a camera stream currently contains the following data: * HDR image rendered by camera * scene depth buffer from camera's point of view #### Overview The whole process of capturing is controlled by DeepDriveCapture which is also responsible of collecting state data from the agent. The actual capturing of the simulated world is done by a Capture Camera Component whereas the processing of a captured snapshot is handled by a Capture Sink. ![Overview](/files/-Lzt9l0VsCm6SHGBkmST) #### Capture Camera Component Capturing the simulated world is done by CaptureCameraComponents. A CaptureCameraComponent inherits from a UCameraComponent and thus represents a camera. CaptureCameraComponents can be arbitrarily placed into the scene and the number of CaptureCameraComponents is not limited. is used to render an HDR image of the simulated world as seen by the camera it represents. It also captures the depth buffer of the scene. A CaptureCameraComponent can simply be added to any Unreal Actor although it will most likely be added to the Actor representing the agent. A TextureRenderTarget2D must be set on a CaptureCameraComponent added to an Actor. The dimension of this render texture defines the dimension of the captured HDR image and depth buffer. The render target must also be defined as HDR: ![Overview](/files/-Lzt9l0XVvqUyVzLd4dy) For each CaptureCameraComponent a camera type can be defined and an user defined id can be set. Additionally a CaptureCameraComponent can be activated or deactivated at runtime by calling *ActivateCapturing()* or *DeactivateCapturing()* from Blueprint or C++. When deactiavted a CaptureCameraComponent isn't contributing to a snapshot anymore. #### Capture Sinks Processing a snapshot is done by a so called CaptureSink. Once a snapshot of the simulated world is taken all snapshot data are forwarded to a CaptureSink. It is possible to have more than one CaptureSink. So there can be a CaptureSink simply saving out the HDR image to disk or a CaptureSink transferring the snapshot's data to a connected client via TCP/IP or any other means of communication. #### Capture Proxy Capturing is internally handled by DeepDriveCapture which is a singleton. DeepDriveCapture is not directly exposed and thus not accessible at least not from UE4's Blueprint system. For that purpose DeepDriveCaptureProxy exists which is an Unreal Engine actor. It exposes some properties such as capture rate to Blueprint. There must be one DeepDriveCaptureProxy actor placed in the scene for capturing to work and if more than one DeepDriveCaptureProxy is placed into the scene (which can not be prevented) the actor on which AActor::PreInitializeComponents() gets called first wins and will become the master DeepDriveCaptureProxy. * detailed description of capture proxy #### BMP Disk capture sink #### Shared memory capture sink ### Control # Plugins # DeepDrivePlugin # Source # DeepDrivePython # LICENSE The MIT License Copyright (c) 2017 DeepDrive () Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # deepdrive\_kevindale Inlined from # Aka Jamestown Moved from # LICENSE ## Deepdrive The MIT License Copyright (c) 2019 Voyage () Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ## TuningCars This work is derived from [TuningCars](https://www.unrealengine.com/marketplace/tuning-cars-kit) on the Unreal Marketplace. Free use, modification, and distribution have been granted by the developer of TuningCars under [this license](https://docs.google.com/document/d/1J3TjLqc1NbXHEeMoppAhcJmSn4q6lJje0Y6DRrh9uHw) for use within the deepdrive/deepdrive-sim repo and its public forks. # Deepdrive sim Unreal based simulator and Python interface to Deepdrive ## Usage Checkout our [main repo](https://github.com/deepdrive/deepdrive) ## Simulator docs [Simulator docs](https://simdocs.deepdrive.io/docs) for making changes to the sim. # Deepdrive Sim Docs ## Setup [Development setup instructions](/v2-security-updates/docs/setup) ## Requirements * Linux * Python 3.6+ * 20GB disk space * 8GB RAM NOTE: Windows support has been removed due to a mismatch of resources and interest. The main impediment to Windows working is the lack of an embedded python interpreter and issues installing the correct packages from uepy-requirements.txt. ## Development Tips ### VERSION file This is used for checking compatibility between sim and agents and will update automatically after building in Unreal. Please check it in. Also if you make a backwards incompatible change, i.e. to the shared memory interface, bump the minor version in the [MAJOR\_MINOR\_VERSION](https://github.com/deepdrive/deepdrive-sim/tree/24e24f3ecaf5f782ef50bffaa23921ae8b5b48fc/docs/Content/Data/MAJOR_MINOR_VERSION/README.md) file. ### Clean builds You'll often want to run `clean.sh` or `clean.bat` after pulling in changes, especially to the plugin as Unreal will spuriously cache old binaries. ### PyCharm If you open an Unreal project in Pycharm, add Binaries, Build, Content, Intermediate, and Saved to your project’s “Excluded” directories in Project Structure or simply by right clicking and choosing “Mark Directory as” => “Excluded”. Keeping these large binary directories in the project will cause PyCharm to index them. Do the same with these directories (Binaries, Build, Content, Intermediate, and Saved) within any of the Plugins in the Plugins folder. Windows - Package via the Unreal Editor ### Setting key binds Unreal->Project Settings->Input->Action Mappings OR in Blueprints->Find (uncheck Find in Current Blueprint Only) and search for the input key, i.e. J. # Setup ## Add yout GitHub account to Epic's GitHub organization First off, [associate your GitHub username with your Unreal account](https://www.unrealengine.com/en-US/ue4-on-github) to get access to Unreal sources on GitHub. ## Set your python bin This is where the Python extension is installed which enables communication between Unreal and Python. ### Option 1: Automatically set the python bin Setup the [deepdrive](https://github.com/deepdrive/deepdrive) project which creates your `~/.deepdrive/python_bin` ### Option 2: Manually set the python bin Create `~/.deepdrive/python_bin` on Unix or `%HOMEPATH%\.deepdrive\python_bin` on Windows and point it to the Python executable you want the deepdrive Python extension to be installed to, i.e. ``` /home/[YOU]/.local/share/virtualenvs/my-env/bin/python ``` ## OS specific setup To finish your development setup, follow one of these guides * [Linux](/v2-security-updates/docs/setup/linux) * [Windows](/v2-security-updates/docs/setup/windows) # Linux Development We support the following local development setups 1. [**Export from prebuilt docker image**](/v2-security-updates/docs/setup/linux/ue4docker-export) (recommended), i.e. ue4-docker installed build - Builds in docker, then installs to your local system. 2. [**Run everything inside docker**](/v2-security-updates/docs/setup/linux/run-in-docker) (promising, but not heavily used) - Uses X11 forwarding view Unreal running in docker 3. [**Traditional**](/v2-security-updates/docs/setup/linux/traditional-install) - Run Unreal per the standard Epic instructions (Use this if you want to use an existing Unreal 4.21 installation) # Running the Unreal Editor from a Docker container ## Contents * [Prerequisites](/v2-security-updates/docs/setup/linux/run-in-docker#prerequisites) * [General instructions](/v2-security-updates/docs/setup/linux/run-in-docker#general-instructions) * [Step 1: Obtaining the Docker image](/v2-security-updates/docs/setup/linux/run-in-docker#step-1-obtaining-the-docker-image) * [Step 2: Starting the container](/v2-security-updates/docs/setup/linux/run-in-docker#step-2-starting-the-container) * [Step 3: Building the project](/v2-security-updates/docs/setup/linux/run-in-docker#step-3-building-the-project) * [Step 4: Running the Editor](/v2-security-updates/docs/setup/linux/run-in-docker#step-4-running-the-editor) * [Step 5: (Optional) Running other commands inside the container](/v2-security-updates/docs/setup/linux/run-in-docker#step-5-optional-running-other-commands-inside-the-container) * [Step 6: Stopping the container](/v2-security-updates/docs/setup/linux/run-in-docker#step-6-stopping-the-container) * [Deepdrive-specific instructions](/v2-security-updates/docs/setup/linux/run-in-docker#deepdrive-specific-instructions) * [Additional prerequisites](/v2-security-updates/docs/setup/linux/run-in-docker#additional-prerequisites) * [Obtaining the Deepdrive Docker image](/v2-security-updates/docs/setup/linux/run-in-docker#obtaining-the-deepdrive-docker-image) * [Additional arguments for Step 2](/v2-security-updates/docs/setup/linux/run-in-docker#additional-arguments-for-step-2) ## Prerequisites In order to run the Unreal Editor from within a Docker container, you will need the following: * A Linux distribution with X11 (and optionally PulseAudio if audio output is required) * The official NVIDIA GPU drivers (open source drivers such as Nouveau will not work) * [Docker](https://www.docker.com/) * The [NVIDIA Docker runtime](https://github.com/NVIDIA/nvidia-docker) ## General instructions ### Step 1: Obtaining the Docker image Before you can start working with an Unreal Editor container, you will first need to build or pull a suitable container image. Voyage employees will be working with the Deepdrive Docker image, which is documented in the section [Obtaining the Deepdrive Docker image](/v2-security-updates/docs/setup/linux/run-in-docker#obtaining-the-deepdrive-docker-image). ### Step 2: Starting the container To start a container from which the Unreal Editor can be run, open a terminal window in the directory containing your Unreal project's source code and execute the following command (**$IMAGE\_NAME** should be replaced with the name of the Docker image that you obtained in Step 1): ```bash docker run --rm -ti --runtime=nvidia --name=unreal_editor "-v$HOME/.config/Epic:/home/ue4/.config/Epic" -v`pwd`:/projectdir -w /projectdir -v/tmp/.X11-unix:/tmp/.X11-unix:rw -e DISPLAY -v/run/user/$UID/pulse:/run/user/1000/pulse $IMAGE_NAME bash ``` The meaning of each of the arguments to the `docker run` command is as follows: * `--rm` ensures the container will be deleted after it has finished running. This ensures you have a clean environment to work from each time you start the image. It is important to note that no data will be lost when the container is deleted, since the project source directory and the Unreal Editor's configuration directory are both bind-mounted from the host system's filesystem. * `-ti` runs the container in interactive mode, which allows you to run commands from the bash shell inside the container in much the same manner as you would on the host system. * `--runtime=nvidia` instructs Docker to use the NVIDIA Docker runtime instead of the default Docker runtime. This allows the container to access the GPU resources of the host system. * `--name=unreal_editor` assigns the identifier "unreal\_editor" to the container. This is useful if you need to run other Docker commands that manipulate or interact with the container while it is running. * `-v$HOME/.config/Epic:/home/ue4/.config/Epic` bind-mounts the host filesystem directory `~/.config/Epic` into the container. This directory is where the Unreal Editor stores its configuration data and compiled shader cache. Storing this data on the host filesystem allows the Unreal Editor to remember information from previous runs and ensures it behaves in a manner more consistent with running the Editor directly on the host system. * ``-v`pwd`:/projectdir`` bind-mounts the current working directory into the container. This allows the Unreal Editor to access your Unreal project's source code. * `-w /projectdir` sets the default working directory of the container to the directory containing your Unreal project's source code. * `-v/tmp/.X11-unix:/tmp/.X11-unix:rw` bind-mounts the host system's X11 socket into the container. This allows the Unreal Editor to create graphical windows that will be displayed on the host system. * `-e DISPLAY` propagates the value of the `DISPLAY` environment variable from the host system to the container. This is needed for correct X11 display behaviour. * `-v/run/user/$UID/pulse:/run/user/1000/pulse` bind-mounts the host system's PulseAudio socket into the container. This is only necessary if audio output is required. * `$IMAGE_NAME` is a placeholder that should be replaced with the name of the Docker image that you obtained in Step 1. * `bash` starts an interactive bash shell within the container. ### Step 3: Building the project Unless you are working with a Blueprint-only Unreal project, you will need to compile your project's C++ source code before you can run the Unreal Editor. You can do this by running the following command inside the container's interactive bash shell: ```bash ue4 build ``` ### Step 4: Running the Editor Once the source code for your Unreal project has been compiled, you can start the Unreal Editor by running the following command inside the container's interactive bash shell: ```bash ue4 run ``` If everything is working correctly, you should see the Unreal Editor splash screen appear while the Editor is loading, followed by the window for the Editor itself. ### Step 5: (Optional) Running other commands inside the container If you would like to run additional commands inside the container while the Unreal Editor is running, you can attach additional interactive bash shells to the container by running the following command: ``` docker exec -ti unreal_editor bash ``` This command can be run multiple times to attach multiple interactive shells. It is important to note that all additional interactive shells will be terminated when the container is stopped, which happens the moment that the original bash shell (the one created when the container was started) exits. Closing any of the additional attached shells will not stop the container. ### Step 6: Stopping the container Once you have closed the Unreal Editor and returned to the container's interactive bash shell, you can stop the container by simply closing the bash shell: ```bash exit ``` Note that this will also terminate any additional interactive shells that you may have attached to the container while it was running. ## Deepdrive-specific instructions ### Additional prerequisites Because the Docker image for Deepdrive contains CUDA 10.0, the version number of the NVIDIA GPU driver matters. You will need to ensure you have a [version of the NVIDIA GPU driver compatible with CUDA 10](https://github.com/NVIDIA/nvidia-docker/wiki/CUDA#requirements). ### Obtaining the Deepdrive Docker image Before you can pull the Docker image for Deepdrive, you will need to login to Docker Hub: ```bash # This will prompt for your Docker ID and password docker login ``` Once you have authenticated, the Docker image for Deepdrive can then be pulled by executing the following command: ```bash docker pull deepdriveio/ue4-deepdrive-deps:latest ``` If you encounter permission issues when pulling the image then you may need to contact Craig Quiter to have your Docker Hub account added to the [deepdriveio organisation](https://hub.docker.com/u/deepdriveio/). ### Additional arguments for Step 2 If you intend to run the Deepdrive deep learning agents inside the container, you will need to bind-mount the directory containing the [Deepdrive source code](https://github.com/deepdrive/deepdrive) in addition to the source code for the Deepdrive simulator Unreal project: ```bash # These are the original arguments specified in Step 2: -v`pwd`:/projectdir -w /projectdir # Replace them with these arguments: -v`pwd`:/simdir -w /simdir -v/path/to/deepdrive:/deepdrivedir ``` Replace `/path/to/deepdrive` with the appropriate host filesystem path for the directory containing the Deepdrive source code. # Export from prebuilt docker image ## Prerequisites * NVIDIA Docker 2 Make sure the ue4-ci-helpers package is up to date ``` pip install --upgrade ue4-ci-helpers pip install ue4-docker ``` ## Build the ue4-full image ## Build the deepdriveio/ue4-deepdrive-deps image ## Export Unreal to your local filesystem ``` ue4-docker export installed deepdriveio/ue4-deepdrive-deps:latest ~/UnrealInstalled ``` Set the default version of Unreal with the [ue4cli](https://pypi.org/project/ue4cli/) ``` ue4 setroot ~/UnrealInstalled ``` ## Build and run the project against the exported UnrealEngine Now you're ready to build and run the project. This is what you'll do any time you've pulled in the latest changes from GitHub as well. Download maps ``` python Packaging/package.py --download-jamestown ``` ### Clean and build Do the following the first time you export or anytime you pull latest from deepdrive-sim. #### Clean your deepdrive-sim project directory to clear any stale build data ``` ./clean_all.sh ``` #### Build and run - takes \~5 minutes ``` ue4 build ue4 run ``` # Traditional Unreal Setup * Clone this repo * Clone our [UnrealEnginePython fork](https://github.com/deepdrive/UnrealEnginePython) into the root of this project (not a submodule as we are going to make this a binary only) ``` cd Plugins git clone https://github.com/deepdrive/UnrealEnginePython ``` * Tip: To avoid rebuilding UnrealEnginePython, move `Plugins/UnrealEnginePython` to your Engine plugins folder, i.e. `/Engine/Plugins/UnrealEnginePython/` after you've built the deepdrive-sim project with the plugin in `deepdrive-sim/Plugins`. ``` git clone git@github.com:EpicGames/UnrealEngine --branch 4.21 # or if you are using http: # git clone https://github.com/EpicGames/UnrealEngine --branch 4.21 ``` `` is now populated with the Unreal Engine source. ## Build Unreal *Takes an hour with 12 cores* ``` cd UnrealEngine ./Setup.sh && ./GenerateProjectFiles.sh && make ``` More details on building Unreal [here](https://wiki.unrealengine.com/Building_On_Linux), though the above commands should be sufficient. ## Get the substance plugin Download the Substance plugin from [here](https://forum.allegorithmic.com/index.php/topic,26732.0.html) and unzip the `Plugins/Runtime/Substance` folder within it to `/Plugins/Runtime` for Unreal 4.21. For other releases, see [here](https://forum.allegorithmic.com/index.php/board,23.0.html) or you can just use the sources downloaded by Windows / Mac marketplace. ## Run the editor ``` ./Engine/Binaries/Linux/UE4Editor ``` Finally open the deepdrive-sim uproject file within the editor. # Windows Development * Use Visual Studio 2015 with the [C++ build tools](https://stackoverflow.com/a/31955339) * Get Unreal v4.21 via the Epic Launcher -> Unreal Engine tab -> Library * Optionally download Unreal sources and debugging symbols in the Epic Launcher ![Windows Unreal Install Options](https://i.imgur.com/Khxc6HV.jpg) * Download UnrealEnginePython \[python binaries] into `Plugins/UnrealEnginePython/Binaries` * Install the Substance Plugin through the Marketplace in Epic Launcher * Open DeepDrive.uproject with the version of Unreal Editor you just installed - if there are errors, check `Saved/Logs` for details * Refresh / Create Visual Studio project * Open the Visual Studio Project * Close Unreal * Right click the DeepDrive project and set as Startup Project, debug... * To run game full speed when the window is not focused * Uncheck Edit->Editor Preferences->Use Less CPU when in Background # Compiling changes to plugins If you make a change to the DeepDrivePlugin, you'll often have to recompile the module to see the changes. To do so in Unreal Editor, open Window->Developer Tools->Modules and search for DeepDrivePlugin. Then hit Recompile to compile your plugin changes. # Vulkan With Unreal 4.21, Vulkan is automatically supported. I've tried it with NVIDIA 384 drivers and things crash, but newer drivers may work. If you experience crashes and see mentions to Vulkan in the logs, you can ensure OpenGL is used by uinstalling these debian packages ``` sudo apt remove libvulkan1 mesa-vulkan-drivers vulkan-utils ``` # python\_bindings ## Building the python bindings ``` install-local-bindings.sh ``` ## Pushing bindings to PyPi Pushing to the release branch causes CI to publish wheels to the PyPi cheese shop. ``` git push origin master && git push origin master:release ``` # Unreal Logging Command line example ``` -LogCmds="global Verbose, LogPython Verbose, LogAnimMontage off, LogDeepDriveAgent VeryVerbose" ``` In DefaultEngine.ini or Engine.ini: ``` [Core.Log] global=[default verbosity for things not listed later] [cat]=[level] foo=verbose break ``` ## Our log categories ``` Defined with DEFINE_LOG_CATEGORY macro ``` ``` LogSunSimulationComponentLogSunLightSimulator LogDeepDriveAgentControllerBase LogDeepDriveAgentSteeringController LogDeepDriveAgent LogDeepDriveSimulationServer LogDeepDriveConnectionThread LogDeepDriveAgentLocalAIController LogDeepDriveAgentSpeedController LogDeepDriveSimulationCaptureProxy LogDeepDriveSimulationMessageHandler LogDeepDriveSimulationServerProxy LogDeepDriveSimulation LogDeepDriveSplineTrack LogSharedMemCaptureMessageBuilder LogSharedMemCaptureSinkWorkerLogSharedMemCaptureSinkWorker LogSharedMemCaptureSinkComponent DeepDriveCaptureProxy DeepDriveCaptureComponent LogDeepDriveCapture LogCaptureBufferPool LogDeepDriveConnectionListener LogDeepDriveClientConnection LogDeepDriveServer LogDeepDrivePlugin LogSharedMemoryImpl_Linux LogPython ``` ## Verbosities Fatal ``` Fatal level logs are always printed to console and log files and crashes even if logging is disabled. ``` Error ``` Error level logs are printed to console and log files. These appear red by default. ``` Warning ``` Warning level logs are printed to console and log files. These appear yellow by default. ``` Display ``` Display level logs are printed to console and log files. ``` Log ``` Log level logs are printed to log files but not to the in-game console. They can still be viewed in editor as they appear via the Output Log window. ``` Verbose ``` Verbose level logs are printed to log files but not the in-game console. This is usually used for detailed logging and debugging. ``` VeryVerbose ``` VeryVerbose level logs are printed to log files but not the in-game console. This is usually used for very detailed logging that would otherwise spam output. ``` # Packaging ## Setup ``` pip install -r Packaging/requirements.txt ``` ## Linux To package a new binary version in Linux run ``` Packaging/package-linux.sh ``` NOTE: Uploads to S3 are done automatically through CI after tests have passed. To upload to S3 anyway: ``` python Packaging/package.py --upload-only ``` ## Windows In Windows we go through the Unreal Editor packaging interface, and manually add UnrealEnginePython. TODO: Fill this out. # Architecture deepdrive-sim consists of two main components. 1. ] The 'game' or simulation, including all the assets, materials, blueprints, lighting, etc... a la a traditional Unreal game project 2. ] The liaison between Unreal and your Python, delivering sensor data from the game and receiving control and configuration commands through a combined C++ Python extension / Unreal Plugin. ## DeepDrivePlugin technical details ### Abstract The DeepDrivePlugin allows extracting images and depth from several cameras via shared memory along with controlling the car and configuring the simulation over TCP sockets. ### Overview The functionality of the DeepDrivePlugin can divided into 2 main domains: * Capturing * Control Capturing provides the means to extract an arbitrary number of live streams out of the simulated world including current state data of the agent. Control provides the means to control capturing. ![Overview](/files/-Lg5Wq5Q5LzdbP5-quI9) ### Capturing Capturing is the process of regularly taking a snapshot of the simulated world and processing the data (i.e. transferring snapshot data to a client). Capturing is separated into these 2 steps: 1. Taking the snapshot 2. Processing the data Currently a snapshot consists of the following data: * state data of agent * upto n camera streams whereas a camera stream currently contains the following data: * HDR image rendered by camera * scene depth buffer from camera's point of view #### Overview The whole process of capturing is controlled by DeepDriveCapture which is also responsible of collecting state data from the agent. The actual capturing of the simulated world is done by a Capture Camera Component whereas the processing of a captured snapshot is handled by a Capture Sink. ![Overview](/files/-Lg5Wq5S8y7WYcIG7Gfm) #### Capture Camera Component Capturing the simulated world is done by CaptureCameraComponents. A CaptureCameraComponent inherits from a UCameraComponent and thus represents a camera. CaptureCameraComponents can be arbitrarily placed into the scene and the number of CaptureCameraComponents is not limited. is used to render an HDR image of the simulated world as seen by the camera it represents. It also captures the depth buffer of the scene. A CaptureCameraComponent can simply be added to any Unreal Actor although it will most likely be added to the Actor representing the agent. A TextureRenderTarget2D must be set on a CaptureCameraComponent added to an Actor. The dimension of this render texture defines the dimension of the captured HDR image and depth buffer. The render target must also be defined as HDR: ![Overview](/files/-Lg5Wq5UnxdwFLAvJh5p) For each CaptureCameraComponent a camera type can be defined and an user defined id can be set. Additionally a CaptureCameraComponent can be activated or deactivated at runtime by calling *ActivateCapturing()* or *DeactivateCapturing()* from Blueprint or C++. When deactiavted a CaptureCameraComponent isn't contributing to a snapshot anymore. #### Capture Sinks Processing a snapshot is done by a so called CaptureSink. Once a snapshot of the simulated world is taken all snapshot data are forwarded to a CaptureSink. It is possible to have more than one CaptureSink. So there can be a CaptureSink simply saving out the HDR image to disk or a CaptureSink transferring the snapshot's data to a connected client via TCP/IP or any other means of communication. #### Capture Proxy Capturing is internally handled by DeepDriveCapture which is a singleton. DeepDriveCapture is not directly exposed and thus not accessible at least not from UE4's Blueprint system. For that purpose DeepDriveCaptureProxy exists which is an Unreal Engine actor. It exposes some properties such as capture rate to Blueprint. There must be one DeepDriveCaptureProxy actor placed in the scene for capturing to work and if more than one DeepDriveCaptureProxy is placed into the scene (which can not be prevented) the actor on which AActor::PreInitializeComponents() gets called first wins and will become the master DeepDriveCaptureProxy. * detailed description of capture proxy #### BMP Disk capture sink #### Shared memory capture sink ### Control # Plugins # DeepDrivePlugin # Source # DeepDrivePython # LICENSE The MIT License Copyright (c) 2017 DeepDrive () Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # deepdrive\_kevindale Inlined from # Aka Jamestown Moved from # LICENSE ## Deepdrive The MIT License Copyright (c) 2016 Deepdrive () Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ## TuningCars This work is derived from [TuningCars](https://www.unrealengine.com/marketplace/tuning-cars-kit) on the Unreal Marketplace. Free use, modification, and distribution have been granted by the developer of TuningCars under [this license](https://docs.google.com/document/d/1J3TjLqc1NbXHEeMoppAhcJmSn4q6lJje0Y6DRrh9uHw) for use within the deepdrive/deepdrive-sim repo and its public forks.