Kopano Groupware Core 8.5.8 Update (2018-04-20):
We have released Kapano Core 8.5.8 after receiving feedback from partners and implementing fixes. The kopano-dbadm tool could not clean the database in multi-server setups with duplicate entries in the “properties” table. The updated version contains fixes for these corner-cases and some improvements on the tool in general.
The packages are now available online.
The steps outlined below and on the wiki page(s) are still valid and should be read carefully before attempting an upgrade to 8.5.8.
The Vulnerabilities
When investigating a report by a customer that a server became slower after an upgrade, our developers discovered this was caused by a large number of items in one of the database tables (the ‘names’ table). Further investigation made clear that this is a vulnerability, which in some cases has a small chance to cause a corruption or data loss in the Kopano Server database.
A client can, within specs, ask the server to store (property name, tag_id) mappings via the getIDsFromNames RPC call. The id is auto generated by the server and monotonically increasing. Due to limitations in the client, it cannot handle values above 31485. The implementation is lacking suitable detection of this condition, and so, because there is no error handling, the client may accidentally access (and overwrite) other properties it did not intend to.
Further details of the vulnerability can be found in tickets KC-1107 and KC-1108 or the CVE entries: CVE-2018-8950 and CVE-2018-8951. The CVE’s are not public yet, and the JIRA-tickets are missing some details. This is done on purpose to give administrators time to upgrade their systems.
The Issue
The issue itself has two components. One is in the code, the other requires a change to the database schema. These changes are automatically applied when upgrading to 8.5.7 and up. When the server (re)starts, it will try to apply the database schema update. This involves a verification of the entries in the “names” table. If this table contains unexpected rows the database schema cannot be updated.
This will block kopano-server from running.
The errors
If the issue of upgrading the database occurs on a system, the administrators will see a combination of the following errors:
K-1216: Cannot update to schema v69, because the "names" table contains unexpected rows. Certain prior versions of the server erroneously allowed these duplicates to be added (KC-1108).
This log entry indicates there is something wrong with the “names” table and the issue blocks the server from starting. K-1216 will always be followed by K-1220 and K-1221, which explain what can be done to resolve the issue(s).
K-1219: It is possible that K-1216 has, in the past, led old clients to misplace data in the DB. This cannot be reliably detected and such data is effectively lost already.
K-1219 only appears when the number of rows can be determined and it exceeds the maximum of 31485. As stated in the error message, misplaced data cannot reliably be detected and is already lost (even before attempting the upgrade).
K-1220: To fix the excess rows, use `kopano-dbadm k-1216`. Consult the manpage and preferably make a backup first.
K-1221: Alternatively, the server may be started with --ignore-da to forego the schema update.
The K-1220 and K-1221 errors give suggestions on how to proceed after upgrading. (‘–ignore-da’ is equivalent to ‘–ignore–database-version-conflict’).
Who is affected?
All previous versions of kopano-server have this vulnerability.
The upgrade issue(s) can appear on any system where kopano-server stores data in an SQL database. Customers with end-users connecting through Outlook will have a higher chance of running into the upgrade issue; without Outlook it is highly unlikely the properties table will have over 30k entries.
kopano-archiver stubbing
Customers running the kopano-archiver service with stubbing enabled have a small chance of running into an issue where stubbed messages only appear in the archive store.
Upgrading to 8.5.7 will fix the issue for stubbing emails from then on.
Resolving the issue
Step-by-step guides that cover various scenario’s have been created on our wiki, below are extracts of these pages and some details on the new kopano-dbadm tool:
N.B. Running the `kopano-dbadm` tool to fix the database requires the kopano-server to be stopped. Make sure you have a database backup before running the tool.
Checking if your installation is affected (before upgrade)
The database can be checked before upgrading with these simple MySQL-queries:
mysql> SELECT MAX(id) from names;
If this returns a value of 31485 or higher, there are too many entries and the database needs to be cleaned.
mysql> select namestring, count(*) as c from names group by guid,nameid,namestring having c>=2;
If this returns any row(s), the database is inconsistent and needs to be cleaned.
After upgrading
To resolve the issues mentioned above, we are releasing a new tool: the Kopano Database Administrator utility.
The kopano-dbadm command line utility will be published in the kopano-utils packages.
This new tool allows administrators to quickly clean up the ‘names’-table when they run into a database with the issue. The tool can also be used to show the fill level of the “names” table.
kopano-dbadm np-stat
Show statistics for the named properties table. This action can be executed while kopano-server is active.
Running the following command will clean up the ‘names’-table by collapsing any duplicates, removing high id’s and defragmenting the table:
kopano-dbadm k-1216
This is an alias for the combination of: np-remove-highid, np-repair-dups, np-defrag. It may be used to address the K-1216 issue that kopano-server(8) may report when performing a schema update.
The clean-up can take a while depending on your database size and hardware configuration.
Workaround
As mentioned in error K-1221, the kopano-server can still be started with the appropriate flag:
kopano-server --ignore-database-version-conflict
This workaround does not prevent future data-loss or corruption, so it is not recommended to run with this flag for a long period of time.
