# DPEcho

Follow this procedure to run DPEcho, a general relativistic magneto hydrodynamics simulator used to model instabilities, turbulence, propagation of waves, stellar winds and magnetospheres, and astrophysical processes around black holes.

- Operating system: Ubuntu* 22.04
- Hardware: Intel® Data Center Max GPUs
- Software: Intel® oneAPI Base toolkit, Intel® oneAPI HPC toolkit
- Time to complete: 15 minutes 

For more information on DPEcho, see [DPEcho description](https://www.intel.com/content/www/us/en/developer/articles/technical/dpecho-general-relativity-sycl-for-2020-beyond.html) and [DPEcho repository](https://github.com/LRZ-BADW/DPEcho).

1. Check whether the driver stack is installed.

   ```(bash)
   $ xpu-smi discovery
   ```
   
   The command should return at least one Intel® Data Center GPU Max device.

2. Check whether the oneAPI toolkit is installed.

   ```(bash)
   $ apt list intel-basekit intel-hpckit
   ```
   
   Expected output:
   ```(bash)
   Listing... Done
   intel-basekit/all,now 2023.2.0-49384 amd64 [installed]
   intel-hpckit/all,now 2023.2.0-49438 amd64 [installed]
   ```

3. If you previously have not configured your environment, install the Ubuntu 22.04 graphics driver.  See [dgpu-docs](https://dgpu-docs.intel.com/) for details.

   ```{note} Access to Ubuntu repositories, such as https://repositories.intel.com and https://apt.repos.intel.com, is required for installation. If proxy settings involve changes to environment variables such as http_proxy or https_proxy, small modifications are required in the following steps, such as adding -E (preserve environment) to sudo commands.
   ```

4. If you previously have not configured your environment, enable access to the Intel repo serving the oneAPI packages and install the oneAPI Base toolkit and HPC toolkit for Ubuntu 22.04.

   ```bash
   wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ | gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
   echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo  tee /etc/apt/sources.list.d/oneAPI.list
   sudo apt update
   ```
   
   ```bash
   sudo apt install -y intel-basekit intel-hpckit
   ```

5. Build DPEcho.

   ```(bash)
   git clone https://github.com/LRZ-BADW/DPEcho.git
   cd DPEcho
   mkdir build && cd build
   source /opt/intel/oneapi/setvars.sh
   cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_MPI=OFF -DFD=6 -DSYCL=oneAPI -DFILE_IO=DISABLED -DSYCL_DEVICE=GPU -DNRK=3 -DPHYSICS=GRMHD -DREC_ORDER=5 -DSINGLE_PRECISION=OFF
   make -j
   ```

6. Run DPEcho.

   ```(bash)
    ./echo_gpu_gr ../examples/alfven.par
   ```

The following example presents a result snapshot from Intel® Data Center GPU Max 1550:

```bash
I:     Raising Hx to allowed min 3
I:     Raising Hy to allowed min 3
I:     Raising Hz to allowed min 3
[0][DeviceConfig::deviceWith] Looking at DPEcho default device.
[0][DeviceConfig::printTargetInfo]
        Hardware Intel(R) Data Center GPU Max 1550GPU
        Max Compute Units  : 896
[0]
        Max Work Group Size: 1024
        Global Memory / GB : 128
        Local  Memory / kB : 128

[0][Domain::cartInfo] CartCoords:    (1919378037 1936749935 1869116255) CartPeriodic:  (1 1 1)
[0][Domain::locInfo] Local size (1 1 1) From (1.91938e+09 1.93675e+09 1.86912e+09) To (1.91938e+09 1.93675e+09 1.86912e+09)
[0][Grid::print]  halos_(3 3 3)  cellsWH(70 70 70)  cellsNH(64 64 64)  Dxyz   (0.015625 0.015625 0.015625)
[0][main] Step 1 out 0 characteristic: 52.049 in 1.97405 s
[0][main] Step 2 out 0 characteristic: 52.0485 in 0.00511193 s
[0][main] Step 3 out 0 characteristic: 52.0486 in 0.00496292 s
[0][main] Step 4 out 0 characteristic: 52.049 in 0.0049932 s
[0][main] Step 5 out 0 characteristic: 52.0483 in 0.00497198 s
[0][main] Step 6 out 0 characteristic: 52.0488 in 0.00499296 s
[0][main] Step 7 out 0 characteristic: 52.049 in 0.0049839 s
[0][main] Step 8 out 0 characteristic: 52.0481 in 0.005023 s
[0][main] Step 9 out 0 characteristic: 52.0489 in 0.00497198 s
[0][main] Step 10 out 0 characteristic: 52.049 in 0.00496912 s
[0][main] Step 11 out 0 characteristic: 52.0479 in 0.00497603 s
[0][main] Step 12 out 0 characteristic: 52.049 in 0.00498796 s
[0][main] Step 13 out 0 characteristic: 52.0489 in 0.0049789 s
[0][main] Step 14 out 0 characteristic: 52.0481 in 0.00497699 s
[0][main] Step 15 out 0 characteristic: 52.0491 in 0.00499701 s
[0][main] Step 16 out 0 characteristic: 52.0488 in 0.00499701 s
[0][main] Step 17 out 0 characteristic: 52.0483 in 0.00499701 s
[0][main] Step 18 out 0 characteristic: 52.0491 in 0.00496602 s
[0][main] Step 19 out 0 characteristic: 52.0487 in 0.00500202 s
[0][main] Step 20 out 0 characteristic: 52.0485 in 0.00496411 s
[0][main] Step 21 out 0 characteristic: 52.0491 in 0.00498676 s
```