To fully benefit from CUDA, PG-Strom requires certain capabilities not provided by mobile chipsets. We have to equip our Mac with full-featured desktop version of an Nvidia GPU.
Thunderbolt to the Rescue
Luckily, most modern Macs are equipped with this awesome new technology, developed by Intel few years go. Its full potential might go unnoticed, as the majority of Apple users associate it almost exclusively with Thunderbolt Displays.
It is much more than a video output, tho. 20 Gb/s of bandwidth (soon to be doubled) makes is fast enough to act as a motherboard extension for powerful extension cards commonly found inside PC towers. For that we’re also going to need a case – my choice is a Sonnet Echo Express III-D.
In my case, it’s going to house a GTX 750 Ti. Not too much of a gaming monster these days, but good enough to give PostgreSQL a noticeable boost. Hardware installation is pretty straightforward, here’s what it looks like put together next to my Mac:
For CUDA acceleration, no display needs to be connected to the card (contrary to 3D gaming).
As I was about to find out at this point, hooking up the cables was the easy bit. Due to the fact that there were no Macs ever to ship with GTX 750 Ti, the card is not supported by OSX. There are so-called “Web Drivers” that Nvidia publishes, but they’re meant to be used with the pro-grade Quadro series.
To make things even more interesting, El Capitan has introduced the almighty System Integrity Protection, which prevents you from fiddling with core OS files even as root. And on top of that, the internals of 5K models are based on logical coupling of 2 display zones (because no single data bus can handle 5120 x 2880 at high refresh rate, apparently).
Alas, here comes a step by step guide on how to make it all work.
Step 1 – Disable SIP
First, check if you have the SIP enabled. It is turned on by default, but you might’ve disabled it in the past when El Cap was introduced because of legacy software issues:
➜ ~ csrutil status System Integrity Protection status: enabled.
There it is – up and running. Let’s try to change that:
➜ ~ csrutil disable csrutil: failed to modify system integrity configuration. This tool needs to be executed from the Recovery OS.
Not so fast – this cannot be done from a running OSX. I have to agree: it does make sense from a security point of view.
To boot into Recovery OS, you have to restart your Mac and hold cmd+R until the apple logo pops up. Once its loaded (it might ask for a language selection), from Utilities menu choose Terminal and try the command again – csrutil disable. You’ll get a message asking you to reboot again – do so.
Step 2 – Download and update the eGPU automate script
Luckily for us, all the dirty work of manually amending kexts and forcing Nvidia drivers to install has been done for us and comes in a form of a single script by goalque.
Let’s put it on the desktop:
➜ ~ cd Desktop ➜ Desktop curl -O https://raw.githubusercontent.com/goalque/automate-eGPU/master/automate-eGPU.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 37925 100 37925 0 0 43047 0 --:--:-- --:--:-- --:--:-- 98251 ➜ Desktop chmod +x automate-eGPU.sh
At this point, if you happen to be using a 5K iMac, you will have to edit this file and change line 62 from this:
config_board_ids=(42FD25EABCABB274 65CE76090165799A B809C3757DA9BB8D DB15BD556843C820 F60DEB81FF30ACF6 FA842E06C61E91C5)
If you don’t, you’re going to find yourself stuck with a 2560 x 2880 resolution (i.e. one vertical plane of the 5K combo).
Step 3 – Run the automator
The script has to be executed as root. Also, make sure your Thunderbolt cable is connected and the Video card powered up (fans spinning) before launching it:
➜ Desktop sudo ./automate-eGPU.sh Password: *** automate-eGPU.sh v0.9.8 - (c) 2015 by Goalque *** ------------------------------------------------------- Detected eGPU GM107 [GeForce GTX 750 Ti] Current OS X 10.11.5 15F34 Previous OS X 10.11.4 15E65 Latest installed Nvidia web driver [not found] No Nvidia web driver detected. Checking IOPCITunnelCompatible keys... Missing IOPCITunnelCompatible keys. Mac board-id found. Searching for matching driver... Driver [346.03.10f02] found from: http://us.download.nvidia.com/Mac/Quadro_Certified/346.03.10f02/WebDriver-346.03.10f02.pkg Do you want to download this driver (y/n)? y % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 70.8M 100 70.8M 0 0 3890k 0 0:00:18 0:00:18 --:--:-- 3418k Driver downloaded. Removing validation checks... Modified package ready. Do you want to install (y/n)? y installer: Package name is NVIDIA Web Driver 346.03.10f02 installer: Installing at base path / installer: The install was successful. installer: The install requires restarting now. Checking IOPCITunnelCompatible keys... Missing IOPCITunnelCompatible keys. IOPCITunnelCompatible mods done. All ready. Please restart the Mac.
It should recognize your GPU, download the Web Drivers from Nvidia, patch & install them, prepare the kexts and ask you to reboot. Do so.
At this point, the card should be visible in System Report:
Back to PostgreSQL
With CUDA-capable card in place, PG-Strom should finally be able to use it. Let’s find out – if you followed my instructions to set up psql, there’s a handy tool called gpuinfo available in your postgres directory:
➜ ~ gpuinfo CUDA Runtime version: 7.5.0 Number of devices: 1 -------- Device Identifier: 0 Device Name: GeForce GTX 750 Ti Global memory size: 2047MB Maximum number of threads per block: 1024 Maximum block dimension X: 1024 Maximum block dimension Y: 1024 Maximum block dimension Z: 64 Maximum grid dimension X: 2147483647 Maximum grid dimension Y: 65535 Maximum grid dimension Z: 65535 Maximum shared memory available per block in bytes: 49152KB Memory available on device for __constant__ variables: 65536bytes Warp size in threads: 32 Maximum number of 32-bit registers available per block: 65536 Typical clock frequency in kilohertz: 1150000KHZ Number of multiprocessors on device: 5 Specifies whether there is a run time limit on kernels: 1 Device is integrated with host memory: false Device can map host memory into CUDA address space: true Compute mode (See CUcomputemode for details): default Device can possibly execute multiple kernels concurrently: true Device has ECC support enabled: false PCI bus ID of the device: 196 PCI device ID of the device: 0 Device is using TCC driver model: false Peak memory clock frequency in kilohertz: 2700000KHZ Global memory bus width in bits: 128 Size of L2 cache in bytes: 2097152bytes Maximum resident threads per multiprocessor: 2048 Number of asynchronous engines: 1 Device shares a unified address space with the host: true Major compute capability version number: 5 Minor compute capability version number: 0 Device supports stream priorities: true Device supports caching globals in L1: false Device supports caching locals in L1: false Maximum shared memory available per multiprocessor: 65536bytes Maximum number of 32bit registers per multiprocessor: 65536 Device can allocate managed memory on this system: false Device is on a multi-GPU board: false Unique id for a group of devices on the same multi-GPU board: 0
It seem we’re in business! Let’s get some data to play with.
I often use a lazy man’s postgres on OSX which comes in a form of a Postgres.APP. One way to grab its databases and put them in our strom-enabled installation is to use pg_upgrade:
➜ ~ ~/postgres/bin/pg_upgrade -d ~/psql -D ~/postgres_data -b /Applications/Postgres.app/Contents/Versions/9.4/bin -B ~/postgres/bin Performing Consistency Checks ----------------------------- Checking cluster versions ok Checking database user is the install user ok Checking database connection settings ok Checking for prepared transactions ok Checking for reg* system OID user data types ok Checking for contrib/isn with bigint-passing mismatch ok Creating dump of global objects ok Creating dump of database schemas ok Checking for presence of required libraries ok Checking database user is the install user ok Checking for prepared transactions ok If pg_upgrade fails after this point, you must re-initdb the new cluster before continuing. Performing Upgrade ------------------ Analyzing all rows in the new cluster ok Freezing all rows on the new cluster ok Deleting files from new pg_clog ok Copying old pg_clog to new server ok Setting next transaction ID and epoch for new cluster ok Deleting files from new pg_multixact/offsets ok Copying old pg_multixact/offsets to new server ok Deleting files from new pg_multixact/members ok Copying old pg_multixact/members to new server ok Setting next multixact ID and offset for new cluster ok Resetting WAL archives ok Setting frozenxid and minmxid counters in new cluster ok Restoring global objects in the new cluster ok Restoring database schemas in the new cluster ok Copying user relation files ok Setting next OID for new cluster ok Sync data directory to disk ok Creating script to analyze new cluster ok Creating script to delete old cluster ok Upgrade Complete ---------------- Optimizer statistics are not transferred by pg_upgrade so, once you start the new server, consider running: ./analyze_new_cluster.sh Running this script will delete the old cluster's data files: ./delete_old_cluster.sh
In this case, ~/psql is my Postgres.APP data directory. pg_upgrade leaves some script files around, which in my opinion could cause some trouble if run by accident:
➜ ~ rm delete_old_cluster.sh ➜ ~ rm analyze_new_cluster.sh
Let’s find out if all these efforts paid off:
➜ ~ pg_ctl -D ~/postgres_data start server starting LOG: PG-Strom version 0.9devel built for PostgreSQL 9.5 LOG: CUDA Runtime version: 7.5.0 LOG: GPU0 GeForce GTX 750 Ti (640 CUDA cores, 1150MHz), L2 2048KB, RAM 2047MB (128bits, 2700MHz), capability 5.0 LOG: NVRTC - CUDA Runtime Compilation vertion 7.5 LOG: database system was shut down at 2016-07-10 14:21:25 EST LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started
That’s it – we have our database engine using a GPU enclosed in a metal box plugged in with a thin cable. What a time to be alive!
Let’s connect to our database to ensure we’re talking to the appropriate daemon:
One last step to complete is to create pg_strom routines in our database:
➜ ~ psql mydb psql (9.5.3) Type "help" for help. mydb=# CREATE EXTENSION pg_strom; CREATE EXTENSION
A quick glance with DataGrip just to make sure:
All systems go! Time to put all these CUDA cores to some use.