The bug took place the 2nd of July 2019 at around 18:00PM.

We apologize to all affected customers. We are making sure it will not happen again.

Service should be back to normal within 4h after this post publish time.

@samotonakatoshi was the first person to give notice about the incident.

Reason

First of all, likwid was not hacked.

The reason was a bad coding practice. To be specific, a bad error handling.

All started when https://dev.steemit.com rpc node gave a random 502 bad request error. Likwid was set to re-invoke itself in such cases. That ended up with multiple Likwid instances running in parallel, which created an exponential endless loop of self-invoking likwid instances. Having that many asynchronous instances made impossible for the database and fail-safe features to prevent multiple redundant payouts.

Final result and lost balance

We found out about the issue quick enough and the server was brought down within the first 20min after the issue-trigger.

The following table contains the different likwid customer accounts who got multiple transfers.

For all affected accounts, we would appreciate if you can send back the lost funds. Please let us know when funds are sent back. Thanks a lot in advance.

Methodology

For the sake of transparency and compliance, please find below the foresincs snippet code that has been used (nodeJS + dsteem library) to identify the lost founds:

async function forensics () {
    var transfers = []
    let history = await client.database.call('get_account_history', ['likwid', -1, 1000])
    var duplicates = []
    transfers = history.filter((x) => {return  (x[1].op[0] == 'transfer' && x[1].op[1].from == 'likwid')}).map((x) => x[1].op[1])
    const unique = [...new Set(transfers.map(item => item.to))]

    for (let i = 0; i < transfers.length; i++) {
        let el = transfers[i]
        let _duplicates = transfers.filter((x) => { return x.to == el.to && x.amount == el.amount})
        if (_duplicates.length > 1) {
            duplicates.push(...new Set(_duplicates))
        }
    }
    let final_duplicates = []
    final_duplicates.push(...new Set(duplicates))
    for (let i = 0; i < unique.length; i++) {
        let author = unique[i]
        let _transfers = final_duplicates.filter((x) => x.to == author)
        if (_transfers.length > 1) {
            console.log(author)
            sbd_transfers = _transfers.filter((x) => getCurrency(x.amount) == 'SBD')
            if (sbd_transfers.length > 1) {
                sbd_transfers.shift()
                total_sbd_debt = sbd_transfers.reduce((a, b) => { return { amount: parseFloat(a.amount) + parseFloat(b.amount) } })
                console.log(parseFloat(total_sbd_debt.amount) + ' SBD')
            }
            steem_transfers = _transfers.filter((x) => getCurrency(x.amount) == 'STEEM')
            if (steem_transfers.length > 1) {
                steem_transfers.shift()
                total_steem_debt = steem_transfers.reduce((a, b) => { return { amount: parseFloat(a.amount) + parseFloat(b.amount) } })
                console.log(parseFloat(total_steem_debt.amount) + ' STEEM')
            }
        }
    }
}