On September 25th 2018, if 17 of the top 20 witnesses (including the current 21st backup witness) are running HF20, the chain will officially hard fork, which will cause any witnesses not running HF20 to be unable to produce blocks. Other nodes such as seeds and RPC nodes may lose connectivity to the network.

As HF20 (v0.20.0) is now available, it's strongly recommended for all witnesses, seeds, and RPC nodes to update prior to the hardfork date (sep 25 2018).

What's new in Steem-in-a-box?

Many improvements have been made to Steem-in-a-box, including an in depth README containing many configuration options for advanced users, such as:

SHM_DIR to place shared_memory somewhere other than /dev/shm
DK_TAG for installing alternative docker images (e.g. older versions, beta versions, custom builds).
REMOTE_WS for overriding the default remote_wallet server (steemd.privex.io)

Other additions:

The logs command now follows the log, no need to keep running it over and over again, or manually typing out the docker logs follow command.
An updated example config for appbase, including a sane default logging config, and seeds changed to the new p2p-seed-node format
Various small fixes

If you're running a previous version of Steem through steem-in-a-box, just CTRL-F/CMD-F for "Upgrading from previous versions".

What's new in HF20?

Everything from the HF19 appbase releases, with some additions. Appbase is part of the hard fork, meaning all developers will need to make their applications appbase ready by September 25th 2018.

See the official release notes on Github.

Disclaimer: HF20 may not be stable yet (as of 14 sep 2018). Please use caution if putting it onto a witness server.

Fresh install

To become a witness, you'll require a server with at LEAST 64GB of RAM, 200GB disk, and a fast, reliable network. I recommend my company @privex - they accept STEEM (and other cryptos), are affordable, and sell servers with a Zero Block Miss SLA

I personally use @privex for my own witness (a top 20) which is proof of their quality. (DISCLAIMER: I'm the CEO of Privex Inc. however you'll find plenty of people here on STEEM unaffiliated with us, praising our servers and network)

You're free to use any other server provider. I don't recommend trying to run a witness on your home internet, as it is likely not reliable enough for a witness.

Be aware that certain server providers have poor networks or hardware, which may cause missed blocks, so you may be hopping around a few until you can find one that's suitable for a witness node.

If you want to learn more about witnesses and what they do, check out my article: https://steemit.com/witness-category/@someguy123/seriously-what-is-a-witness-why-should-i-care-how-do-i-become-one-answer

Shared memory note

As of today (Fri 14th September 2018), it's recommended for witnesses/seeds to have at least 64 gigabytes (GB) of RAM. This is because Steem stores information in a file called "shared_memory.bin" which is used similar to RAM. This file is around 44gb on HF20, and will grow over time, so 64gb will give you 10-15gb of room for this to grow.

Install steem-docker and dependencies

First, we install some basic dependencies (assuming you're on Ubuntu/Debian).

sudo apt update
sudo apt install git curl wget

Next, we install steem-docker (a line starting with a # is a comment, you don't need to paste them)

git clone https://github.com/someguy123/steem-docker.git
cd steem-docker
# If you don't already have a docker installation, this will install it for you
./run.sh install_docker

After the install_docker command has finished, it's recommended to log out and back in again (e.g. close your SSH client and re-connect) to ensure that the shell correctly detects docker.

Then, you can download and install the latest witness/seed docker image

# This downloads/updates the low-memory docker image for Steem
./run.sh install

If you are a witness, you need to adjust the configuration as needed e.g. witness name, private key, logging config, turn off p2p-endpoint etc.

If you're running a seed, then don't worry about the config, it will just work and you can skip the signing key / config section

Generating a signing key

For witnesses, you'll need to generate a signing key first. You can use the remote wallet for this.

Open the wallet using a public server with the following command:

./run.sh remote_wallet

Note: you may receive an error such as "timeout" or "invalid http status". This is normal, it's usually due to a server in the Privex load balancer having temporary problems.

Just run the command again a few times until it connects (you'll see new>>> or locked>>>)

Create a key pair for your witness.

suggest_brain_key

You should get something similar to this:

  "wif_priv_key": "5xxxxxxxxxxxxxxxxxxx",
  "pub_key": "STMxxxxxxxxxxxxxxxxx"

Put the keys in notepad or something similar for the moment so you don't lose them.

Press CTRL-D to exit the wallet.

Setting up your config

Now, to set your witness name and private key. You can use nano (a simple text editor on Linux) for this.

nano data/witness_node_data_dir/config.ini

If you are running a witness, you'll want to change one of the first lines in the file to disable the seed.

# Endpoint for P2P node to listen on
p2p-endpoint = 0.0.0.0:2001

Simply place a # (hash sign) at the start of the p2p-endpoint line. This will disable the seed. Example:

# Endpoint for P2P node to listen on
#p2p-endpoint = 0.0.0.0:2001

Now to add your witness settings. Scroll down the file (use the up/down arrows, or page up/down) until you see something like this:

# Percent of witnesses (0-99) that must be participating in order to produce blocks
required-participation = 33

# name of witness controlled by this node (e.g. initwitness )
# witness =

# name of miner and its private key (e.g. ["account","WIF PRIVATE KEY"] )
# miner =

# Number of threads to use for proof of work mining
# mining-threads =

# WIF PRIVATE KEY to be used by one or more witnesses or miners
# private-key =

You'll want to uncomment (remove the # from the start) the witness = line, and the private-key = line.

Grab your public/private key that you saved earlier into notepad (or other similar app).

Then, similar to below, add your witness' name in quotes, e.g. "someguy123" in front of the witness =, and your private key (without quotes) in front of the private-key = line. To save you time in the future, it can be useful to place your public signing key in a comment above your private key (see example)

Example:

# name of witness controlled by this node (e.g. initwitness )
witness = "someguy123"

# name of miner and its private key (e.g. ["account","WIF PRIVATE KEY"] )
# miner =

# Number of threads to use for proof of work mining
# mining-threads =

# WIF PRIVATE KEY to be used by one or more witnesses or miners
# STMxxxxxxxxxx
private-key = 5xxxxxxxxx

Now (assuming you're using nano) press CTRL-X, and press the y key when it asks if you would like to save.

Environment settings

If you're already comfortable with Linux, and want to play around with settings for steem-docker you may want to consider setting up a .env file, see the README for more information

Otherwise, let's continue. The defaults are generally fine for most people.

Setting up shared memory, swap settings, and downloading the blockchain

At this point you should have:

installed steem-in-a-box
installed docker (through the run.sh command)
downloaded and installed the steem docker container
set up your witness configuration

Your witness is almost ready. Just a few more small things.

You'll need to set the shared memory size (use sudo if not logged in as root). As of today (Sep 14th 2018), 64 gigabytes is recommended (64G). Please be aware that the shared memory size changes constantly. Ask in the witness chatroom if you're unsure.

sudo ./run.sh shm_size 64G

You'll also want to adjust your "swappiness". This is a setting which tells Linux how much it should use swap (virtual memory on your disk. very slow). You'll want to set this to 1, which means "don't use the swap unless it's absolutely necessary". This helps to prevent missed blocks due to excessive swapping despite plenty of free RAM.

# To persist on reboot, place 'vm.swappiness=1' in /etc/sysctl.conf
sudo sysctl -w vm.swappiness=1

Now that you've configured your server, it's recommended to download the block log, as replays can be faster than p2p download.

./run.sh dlblocks

Almost ready!

Once this is done, you should be ready to start your server. The replay command will start steemd in docker, and begin replaying the blockchain which was downloaded to create a block index, and the shared_memory file.

./run.sh replay

You can check the status with the logs command.

./run.sh logs

Assuming you did everything correctly, you should see something like this:

------------------------------------------------------

            STARTING STEEM NETWORK

------------------------------------------------------
initminer public key: STM8GC13uCZbP44HzMLV6zPZGwVQ8Nt4Kji8PapsPiNq1BK153XTX
chain id: 0000000000000000000000000000000000000000000000000000000000000000
blockchain version: 0.20.0
------------------------------------------------------
2393837ms main.cpp:121                  main                 ] Backtrace on segfault is enabled.
2393837ms chain_plugin.cpp:405          plugin_startup       ] Starting chain with shared_file_size: 53687091200 bytes
2393837ms chain_plugin.cpp:483          plugin_startup       ] Replaying blockchain on user request.
2393837ms database.cpp:185              reindex              ] Reindexing Blockchain
2396435ms block_log.cpp:142             open                 ] Log is nonempty
2396435ms block_log.cpp:151             open                 ] Index is nonempty
2396435ms database.cpp:193              reindex              ] Replaying blocks...

Please be aware that this can take many hours depending on your disk speed and CPU. In some cases it can take up to 24 hours.

You can press CTRL-C to exit the logs, or just leave it if you want to watch it replay.

After you've replayed

Once you see something which looks like this, it means your server is now ready to go:

1299055ms th_a       application.cpp:507           handle_block         ] Got 14 transactions on block 12928269 by pharesim -- latency: 55 ms
1302427ms th_a       application.cpp:507           handle_block         ] Got 18 transactions on block 12928270 by xeldal -- latency: 426 ms
1305291ms th_a       application.cpp:507           handle_block         ] Got 26 transactions on block 12928271 by arhag -- latency: 291 ms
1308045ms th_a       application.cpp:507           handle_block         ] Got 20 transactions on block 12928272 by pfunk -- latency: 45 ms
1311092ms th_a       application.cpp:507           handle_block         ] Got 23 transactions on block 12928273 by bhuz -- latency: 92 ms

Go to your profile on Steemit, click Wallet, then Permissions. Next to "Active Key" you should see "Login to show". Click that button and log in. Then press "Show private key", to reveal it. Make sure it begins with a "5".

Copy the key.

Now open the wallet with:

./run.sh wallet

(Image courtesy of @klye, copied from his old guide)

Set a password, unlock the wallet with the same password, then import your ACTIVE private key from steemit (NOT the one in notepad):

set_password "mysupersecurepass"
unlock "mysupersecurepass"
import_key 5zzzzzzzzzzzz

Finally, we can update/create the witness on the network. Replace the STMxxxx key with the public key you saved earlier in notepad (or if you wrote it as a comment in your config file, you can open that and get it from there), and replace YOURNAME with your witness name on Steemit.

update_witness "YOURNAME" "https://steemit.com/witness-category/@YOURNAME/my-witness-thread" "STMxxxxxxx" {"account_creation_fee":"0.100 STEEM","maximum_block_size":65536,"sbd_interest_rate":0} true

Congratulations, your witness should now be working, and you can begin getting votes and producing blocks

Other useful information

You may want to persist the /dev/shm size (shared memory) across reboots. To do this, you can edit /etc/fstab, please be very careful, as any mistakes in this file will cause your system to become unbootable. (if you are not comfortable with linux, please don't try this as you may break your server).

Simply add this to the bottom of the file on a new line. Be sure not to damage any other lines in the file. Adjust "64G" to whatever size you would like /dev/shm to be.

tmpfs   /dev/shm         tmpfs   nodev,nosuid,size=64G          0  0

See the README for more advanced options.

Upgrading from previous versions

Please be warned. Upgrading to HF20 will require a replay (even if you're on an appbase release). It's strongly recommended to do this on your backup server first, or change to your backup server during the upgrade.

Server Version Check

First, you'll want to check your server's version.

This is as simple as:

docker logs seed | grep "blockchain version"

(if you have renamed your docker container, you may need to change seed to witness etc.)

You should see something like:

blockchain version: 0.19.6

If your version is lower than 0.19.10, then you're not on appbase, and will have to re-make your config.

If you're already on appbase (anything higher than 0.19.10), you can skip the config update parts.

Upgrade procedure

Again, upgrading to HF20 will require a replay. Do this on your backup server, or disable your witness during the upgrade!

First, update your steem-docker files (this will make sure you're on the master branch, and not the old appbase branch)

~/steem-docker $ git fetch
~/steem-docker $ git checkout master
~/steem-docker $ git pull

Move your old config out of the way. Appbase has a new config format, Steem-in-a-box will create a new example config for you.

mv data/witness_node_data_dir/config.ini data/witness_node_data_dir/config.ini.bak-sep2018

Download the HF20 docker image using this command. Assuming you've renamed your config file, this will also generate a new example config for you into config.ini.

./run.sh install

Fixing your config

Now, you'll want to put your witness name and key back into the new config.

This command will quickly obtain your witness name and private key for copy pasting:

cat data/witness_node_data_dir/config.ini.bak-sep2018 | grep -E "^witness|private-key"

You'll see something like this:

~/steem-docker $ cat data/witness_node_data_dir/config.ini.bak-sep2018 | grep -E "^witness|private-key"
witness = "someguy123"
private-key = 5xxxxxxxxxxxxxxxxx

Open your new config:

nano data/witness_node_data_dir/config.ini

Comment out p2p-endpoint (put a # at the start of it) if this is for a witness. You don't want your witness node running a seed.

# p2p-endpoint = 0.0.0.0:2001

Place your witness name and private key somewhere in the middle of the file (near the "name of witness controlled" comment makes it easy to find), like this:

# Percent of witnesses (0-99) that must be participating in order to produce blocks
required-participation = 33

# name of witness controlled by this node (e.g. initwitness )
# witness =
witness = "someguy123"
private-key = 5xxxxxxxxxxxxxxxxx

For the shared memory settings at the top, I strongly recommend leaving it at 60G, as it will automatically expand (new appbase feature) when it needs to.

A server with 64gb RAM is still the recommended size for witnesses, however this may change over the next few weeks as the chain grows.

Now for the slow part - replaying

When upgrading from prior versions to HF20, you'll need to replay.

To replay your witness, do the following (this will shutdown and remove the pre-appbase software, and start an appbase one in replay mode)

./run.sh stop
./run.sh replay

You can check the status of the replay with the logs command

./run.sh logs

Note: the command now automatically updates in realtime. to exit the logs press CTRL-C

If all is well, you'll see it replaying on v0.20.0 (or whatever version is out by the time you read this):

------------------------------------------------------

            STARTING STEEM NETWORK

------------------------------------------------------
initminer public key: STM8GC13uCZbP44HzMLV6zPZGwVQ8Nt4Kji8PapsPiNq1BK153XTX
chain id: 0000000000000000000000000000000000000000000000000000000000000000
blockchain version: 0.20.0
------------------------------------------------------
2393837ms main.cpp:121                  main                 ] Backtrace on segfault is enabled.
2393837ms chain_plugin.cpp:405          plugin_startup       ] Starting chain with shared_file_size: 53687091200 bytes
2393837ms chain_plugin.cpp:483          plugin_startup       ] Replaying blockchain on user request.
2393837ms database.cpp:185              reindex              ] Reindexing Blockchain
2396435ms block_log.cpp:142             open                 ] Log is nonempty
2396435ms block_log.cpp:151             open                 ] Index is nonempty
2396435ms database.cpp:193              reindex              ] Replaying blocks...

Once it's replayed (may take 3 or more hours), you'll see the standard "got x transactions on block y".

1668107ms p2p_plugin.cpp:210            handle_block         ] Got 54 transactions on block 25299003 by cervantes -- latency: 107 ms
1671090ms p2p_plugin.cpp:210            handle_block         ] Got 36 transactions on block 25299004 by blocktrades -- latency: 90 ms
1674098ms p2p_plugin.cpp:210            handle_block         ] Got 28 transactions on block 25299005 by roelandp -- latency: 98 ms

If you see this, you can now re-enable this server :)

You're successfully upgraded to 0.20.0 (or higher).

For future HF20 releases, it will now be as simple as ./run.sh install && ./run.sh restart, but as always, be careful, always assume a replay may be needed. Test on your backup, or disable your witness before attempting to update.

To see if I've released a new version of the docker image and haven't made a post yet, just check my docker hub: https://hub.docker.com/r/someguy123/steem/tags/

GIF Avatar by @stellabelle

Do you like what I'm doing for STEEM/Steemit?

Vote for me to be a witness - every vote counts.

Don't forget to follow me for more like this.

Have you ever thought about being a witness yourself? Join the witness channel. We're happy to guide you! Join in shaping the STEEM economy.

Are you looking for a new server provider? My company @privex offers highly-reliable and affordable dedicated and virtual servers for STEEM, LTC, and BTC! Check out our website at https://www.privex.io

The easy way to install or upgrade to Hardfork 20 (Steem-in-a-box)