From 2cad7a67ead35ce72a0b6635f70a84da0ed43b27 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20B=C3=B6hmer?= Date: Fri, 21 Jun 2024 10:39:44 +0200 Subject: [PATCH 1/4] Updated requirements in README file and mention PostgreSQL support in features --- README.md | 8 ++++---- docs/index.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 023dd012..74ebfe7f 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ for the first time. * Event log: Track what changes happen to your inventory, track which user does what. Revert your parts to older versions. * Responsive design: You can use Part-DB on your PC, your tablet, and your smartphone using the same interface. -* MySQL and SQLite are supported as database backends +* MySQL, SQLite and PostgreSQL are supported as database backends * Support for rich text descriptions and comments in parts * Support for multiple currencies and automatic update of exchange rates supported * Powerful search and filter function, including parametric search (search for parts according to some specifications) @@ -74,10 +74,10 @@ Part-DB is also used by small companies and universities for managing their inve ## Requirements * A **web server** (like Apache2 or nginx) that is capable of - running [Symfony 5](https://symfony.com/doc/current/reference/requirements.html), + running [Symfony 6](https://symfony.com/doc/current/reference/requirements.html), this includes a minimum PHP version of **PHP 8.1** -* A **MySQL** (at least 5.7) /**MariaDB** (at least 10.2.2) database server if you do not want to use SQLite. -* Shell access to your server is highly suggested! +* A **MySQL** (at least 5.7) /**MariaDB** (at least 10.4) database server, or **PostgreSQL** 10+ if you do not want to use SQLite. +* Shell access to your server is highly recommended! * For building the client-side assets **yarn** and **nodejs** (>= 18.0) is needed. ## Installation diff --git a/docs/index.md b/docs/index.md index 7dafabf5..d732f31d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -41,7 +41,7 @@ It is installed on a web server and so can be accessed with any browser without * Event log: Track what changes happens to your inventory, track which user does what. Revert your parts to older versions. * Responsive design: You can use Part-DB on your PC, your tablet and your smartphone using the same interface. -* MySQL and SQLite supported as database backends +* MySQL, SQLite and PostgreSQL are supported as database backends * Support for rich text descriptions and comments in parts * Support for multiple currencies and automatic update of exchange rates supported * Powerful search and filter function, including parametric search (search for parts according to some specifications) From 22855b077d28e8e0a8e7de9a83bceea46b1e143c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20B=C3=B6hmer?= Date: Fri, 21 Jun 2024 11:44:24 +0200 Subject: [PATCH 2/4] Improved "Choosing database" documentation page --- docs/installation/choosing_database.md | 102 +++++++++++++++++++------ 1 file changed, 79 insertions(+), 23 deletions(-) diff --git a/docs/installation/choosing_database.md b/docs/installation/choosing_database.md index 99d5f886..a59fef39 100644 --- a/docs/installation/choosing_database.md +++ b/docs/installation/choosing_database.md @@ -7,10 +7,18 @@ nav_order: 1 # Choosing database: SQLite or MySQL -Part-DB saves its data in a [relational (SQL) database](https://en.wikipedia.org/wiki/Relational_database). Part-DB -supports either the use of [SQLite](https://www.sqlite.org/index.html) -or [MySQL](https://www.mysql.com/) / [MariaDB](https://mariadb.org/) (which are mostly the same, except for some minor -differences). +Part-DB saves its data in a [relational (SQL) database](https://en.wikipedia.org/wiki/Relational_database). + +For this multiple database types are supported, currently these are: + +* [SQLite](https://www.sqlite.org/index.html) +* [MySQL](https://www.mysql.com/) / [MariaDB](https://mariadb.org/) (which are mostly the same, except for some minor + differences) +* [PostgreSQL](https://www.postgresql.org/) + +All these database types allow for the same basic functionality and allow Part-DB to run. However, there are some minor +differences between them, which might be important for you. Therefore the pros and cons of the different database types +are listed here. {: .important } You have to choose between the database types before you start using Part-DB and **you can not change it (easily) after @@ -18,29 +26,77 @@ you have started creating data**. So you should choose the database type for you ## Comparison -**SQLite** is the default database type which is configured out of the box. All data is saved in a single file ( -normally `var/app.db` in the Part-DB folder) and no additional installation or configuration besides Part-DB is needed. -To use **MySQL/MariaDB** as database, you have to install and configure the MySQL server, configure it and create a -database and user for Part-DB, which needs some additional work. When using docker you need an additional docker -container, and volume for the data +### SQLite -When using **SQLite** The database can be backuped easily by just copying the SQLite file to a safe place. Ideally, the * -*MySQL** database has to be dumped to a SQL file (using `mysqldump`). The `console partdb:backup` command can do this -automatically +#### Pros -However, SQLite does not support certain operations like regex search, which has to be emulated by PHP and therefore is -pretty slow compared to the same operation at MySQL. In the future, there might be features that may only be available, when -using MySQL. Also, SQLite has limitations in comparisons and sorting of Unicode characters, which might lead to unexpected -behavior when using non-ASCII characters in your data. For example `µ` (micro sign) is not seen as equal to `μ(greek minuscule mu), -therefore searching for `µ` (micro sign) will not find parts containing `μ` (mu) and vice versa. In MySQL identical-looking characters are seen as equal, which is more intuitive in most cases. +* **Easy to use**: No additional installation or configuration is needed, just start Part-DB and it will work out of the box +* **Easy backup**: Just copy the SQLite file to a safe place, and you have a backup, which you can restore by copying it + back. No need to work with SQL dumps -In general MySQL might perform better for big Part-DB instances with many entries, lots of users and high activity, than -SQLite. +#### Cons -## Conclusion and Suggestion +* **Performance**: SQLite is not as fast as MySQL or PostgreSQL, especially when using complex queries or many users. +* **Emulated RegEx search**: SQLite does not support RegEx search natively. Part-DB can emulate it, however that is pretty slow. +* **Emualted natural sorting**: SQLite does not support natural sorting natively. Part-DB can emulate it, but it is pretty slow. +* **Limitations with Unicode**: SQLite has limitations in comparisons and sorting of Unicode characters, which might lead to + unexpected behavior when using non-ASCII characters in your data. For example `µ` (micro sign) is not seen as equal to + `μ` (greek minuscule mu), therefore searching for `µ` (micro sign) will not find parts containing `μ` (mu) and vice versa. + The other databases behave more intuitive in this case. +* **No advanced features**: SQLite do no support many of the advanced features of MySQL or PostgreSQL, which might be utilized + in future versions of Part-DB + + +### MySQL/MariaDB + +**If possible, it is recommended to use MariaDB 10.7+ (instead of MySQL), as it supports natural sorting of columns natively.** + +#### Pros + +* **Performance**: Compared to SQLite, MySQL/MariaDB will probably perform better, especially in large databases with many + users and high activity. +* **Natural Sorting**: MariaDB 10.7+ supports natural sorting of columns. On other databases it has to be emulated, which is pretty + slow. +* **Native RegEx search**: MySQL supports RegEx search natively, which is faster than emulating it in PHP. +* **Advanced features**: MySQL/MariaDB supports many advanced features, which might be utilized in future versions of Part-DB. +* **Full Unicode support**: MySQL/MariaDB has better support for Unicode characters, which makes it more intuitive to use + non-ASCII characters in your data. + +#### Cons + +* **Additional installation and configuration**: You have to install and configure the MySQL server, create a database and + user for Part-DB, which needs some additional work compared to SQLite. +* **Backup**: The MySQL database has to be dumped to a SQL file (using `mysqldump`). The `console partdb:backup` command can automate this. + + +### PostgreSQL + +#### Pros +* **Performance**: PostgreSQL is known for its performance, especially in large databases with many users and high activity. +* **Advanced features**: PostgreSQL supports many advanced features, which might be utilized in future versions of Part-DB. +* **Full Unicode support**: PostgreSQL has better support for Unicode characters, which makes it more intuitive to use + non-ASCII characters in your data. +* **Native RegEx search**: PostgreSQL supports RegEx search natively, which is faster than emulating it in PHP. +* **Native Natural Sorting**: PostgreSQL supports natural sorting of columns natively in all versions and in general the support for it + is better than on MariaDB. +* **Support of transactional DDL**: PostgreSQL supports transactional DDL, which means that if you encounter a problem during a schema change, +the database will automatically rollback the changes. On MySQL/MariaDB you have to manually rollback the changes, by restoring from a database backup. + +#### Cons +* **New backend**: The support of postgresql is new, and it was not tested as much as the other backends. There might be some bugs caused by this. +* **Additional installation and configuration**: You have to install and configure the PostgreSQL server, create a database and + user for Part-DB, which needs some additional work compared to SQLite. +* **Backup**: The PostgreSQL database has to be dumped to a SQL file (using `pg_dump`). The `console partdb:backup` command can automate this. + + +## Recommendation When you are a hobbyist and use Part-DB for your own small inventory management with only you as user (or maybe sometimes -a few other people), then the easy-to-use SQLite database will be fine. +a few other people), then the easy-to-use SQLite database will be fine, as long as you can live with the limitations, stated above. +However using MariaDB (or PostgreSQL), has no disadvantages in that situation (besides the initial setup requirements), so you might +want to use it, to be prepared for future use cases. -When you are planning to have a very big database, with a lot of entries and many users which regularly (and -concurrently) using Part-DB you should maybe use MySQL as this will scale better. \ No newline at end of file +When you are planning to have a very big database, with a lot of entries and many users which regularly using Part-DB, then you should +use MariaDB or PostgreSQL, as they will perform better in that situation and allow for more advanced features. +If you should use MariaDB or PostgreSQL depends on your personal preference and what you already have installed on your servers and +what you are familiar with. \ No newline at end of file From 2fabcabcc0dab4d84382e469ea5d857d17a8c23f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20B=C3=B6hmer?= Date: Fri, 21 Jun 2024 11:58:09 +0200 Subject: [PATCH 3/4] Added documentation about the different DATABASE_URL formats for the database types --- docs/installation/choosing_database.md | 56 +++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) diff --git a/docs/installation/choosing_database.md b/docs/installation/choosing_database.md index a59fef39..2f9002bc 100644 --- a/docs/installation/choosing_database.md +++ b/docs/installation/choosing_database.md @@ -99,4 +99,58 @@ want to use it, to be prepared for future use cases. When you are planning to have a very big database, with a lot of entries and many users which regularly using Part-DB, then you should use MariaDB or PostgreSQL, as they will perform better in that situation and allow for more advanced features. If you should use MariaDB or PostgreSQL depends on your personal preference and what you already have installed on your servers and -what you are familiar with. \ No newline at end of file +what you are familiar with. + +## Using the different databases + +The only difference in using the different databases, is a different value in the `DATABASE_URL` environment variable in the `.env.local` file +or in the `DATABASE_URL` environment variable in your server or container configuration. It has the shape of a URL, where the scheme (the part before `://`) +is the database type, and the rest is connection information. + +**The env var format below is for the `env.local` file. It might work differently for other env configuration. E.g. in a docker-compose file you have to remove the quotes!** + +### SQLite + +```shell +DATABASE_URL="sqlite:///%kernel.project_dir%/var/app.db" +``` + +Here you just need to configure the path to the SQLite file, which is created by Part-DB when performing the database migrations. +The `%kernel.project_dir%` is a placeholder for the path to the project directory, which is replaced by the actual path by Symfony, so that you do not +need to specify the path manually. In the example the database will be created as `app.db` in the `var` directory of your Part-DB installation folder. + +### MySQL/MariaDB + +```shell +DATABASE_URL="mysql://user:password@127.0.0.1:3306/database?serverVersion=8.0.37" +``` + +Here you have to replace `user`, `password` and `database` with the credentials of the MySQL/MariaDB user and the database name you want to use. +The host (here 127.0.0.1) and port should also be specified according to your MySQL/MariaDB server configuration. + +In the `serverVersion` parameter you can specify the version of the MySQL/MariaDB server you are using, in the way the server returns it +(e.g. `8.0.37` for MySQL and `10.4.14-MariaDB`). If you do not know it, you can leave the default value. + +If you want to use a unix socket for the connection instead of a TCP connnection, you can specify the socket path in the `unix_socket` parameter. +```shell +DATABASE_URL="mysql://user:password@localhost/database?serverVersion=8.0.37&unix_socket=/var/run/mysqld/mysqld.sock" +``` + +### PostgreSQL + +```shell +DATABASE_URL="postgresql://db_user:db_password@127.0.0.1:5432/db_name?serverVersion=12.19&charset=utf8" +``` + +Here you have to replace `db_user`, `db_password` and `db_name` with the credentials of the PostgreSQL user and the database name you want to use. +The host (here 127.0.0.1) and port should also be specified according to your PostgreSQL server configuration. + +In the `serverVersion` parameter you can specify the version of the PostgreSQL server you are using, in the way the server returns it +(e.g. `12.19 (Debian 12.19-1.pgdg120+1)`). If you do not know it, you can leave the default value. + +The `charset` parameter specify the character set of the database. It should be set to `utf8` to ensure that all characters are stored correctly. + +If you want to use a unix socket for the connection instead of a TCP connnection, you can specify the socket path in the `unix_socket` parameter. +```shell +DATABASE_URL="postgresql://db_user:db_password@localhost/db_name?serverVersion=12.19&charset=utf8&unix_socket=/var/run/postgresql/.s.PGSQL.5432" +``` From 1f6e3db09ee738d49579d13925d04c8ffe0a5a4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20B=C3=B6hmer?= Date: Fri, 21 Jun 2024 12:31:25 +0200 Subject: [PATCH 4/4] Document natural sorting and the DATABASE_EMULATE_NATURAL_SORT option --- .env | 2 +- docs/configuration.md | 3 +++ docs/installation/choosing_database.md | 26 ++++++++++++++++++++++++++ 3 files changed, 30 insertions(+), 1 deletion(-) diff --git a/.env b/.env index b35b17f5..cf39b10d 100644 --- a/.env +++ b/.env @@ -24,7 +24,7 @@ DATABASE_MYSQL_USE_SSL_CA=0 DATABASE_MYSQL_SSL_VERIFY_CERT=1 # Emulate natural sorting of strings even on databases that do not support it (like SQLite, MySQL or MariaDB < 10.7) -# This can be slow on big databases +# This can be slow on big databases and might have some problems and quirks, so use it with caution DATABASE_EMULATE_NATURAL_SORT=0 ################################################################################### diff --git a/docs/configuration.md b/docs/configuration.md index 8990e9a7..c0daa407 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -40,6 +40,9 @@ options listed, see `.env` file for the full list of possible env variables. * `DATABASE_MYSQL_USE_SSL_CA`: If this value is set to `1` or `true` and a MySQL connection is used, then the connection is encrypted by SSL/TLS and the server certificate is verified against the system CA certificates or the CA certificate bundled with Part-DB. Set `DATABASE_MYSQL_SSL_VERIFY_CERT` if you want to accept all certificates. +* `DATABASE_EMULATE_NATURAL_SORT` (default 0): If set to 1, Part-DB will emulate natural sorting, even if the database + does not support it natively. However this is much slower than the native sorting, and contain bugs or quirks, so use + it only, if you have to. * `DEFAULT_LANG`: The default language to use server-wide (when no language is explicitly specified by a user or via language chooser). Must be something like `en`, `de`, `fr`, etc. * `DEFAULT_TIMEZONE`: The default timezone to use globally, when a user has no timezone specified. Must be something diff --git a/docs/installation/choosing_database.md b/docs/installation/choosing_database.md index 2f9002bc..69dc810e 100644 --- a/docs/installation/choosing_database.md +++ b/docs/installation/choosing_database.md @@ -154,3 +154,29 @@ If you want to use a unix socket for the connection instead of a TCP connnection ```shell DATABASE_URL="postgresql://db_user:db_password@localhost/db_name?serverVersion=12.19&charset=utf8&unix_socket=/var/run/postgresql/.s.PGSQL.5432" ``` + + +## Natural Sorting + +Natural sorting is the sorting of strings in a way that numbers are sorted by their numerical value, not by their ASCII value. + +For example in the classical binary sorting the string `DIP-4`, `DIP-8`, `DIP-16`, `DIP-28` would be sorted as following: + +* `DIP-16` +* `DIP-28` +* `DIP-4` +* `DIP-8` + +In natural sorting, it would be sorted as: + +* `DIP-4` +* `DIP-8` +* `DIP-16` +* `DIP-28` + +Part-DB can sort names in part tables and tree views naturally. PostgreSQL and MariaDB 10.7+ support natural sorting natively, +and it is automatically used if available. + +For SQLite and MySQL < 10.7 it has to be emulated if wanted, which is pretty slow. Therefore it has to be explicity enabled by setting the +`DATABASE_EMULATE_NATURAL_SORT` environment variable to `1`. If it is 0 the classical binary sorting is used, on these databases. The emulations +might have some quirks and issues, so it is recommended to use a database which supports natural sorting natively, if you want to use it.