Compare commits
3 Commits
60f6a85bb2
...
6d66a7be70
| Author | SHA1 | Date |
|---|---|---|
 Nicolas Massé
|
6d66a7be70 | 4 days ago |
|
Nicolas Massé
|
ab9040dce2 | 5 days ago |
|
Nicolas Massé
|
d33bfdc0d2 | 5 days ago |
11 changed files with 390 additions and 6 deletions
@ -0,0 +1,13 @@ |
|||
# Podman Quadlet Cookbook |
|||
|
|||
[Podman Quadlets](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) are awesome, but vastly under-utilized in the Open Source communities. |
|||
This repository gathers all the recipes (hence the name "Cookbook") to deploy Open Source technologies using Podman Quadlets. |
|||
|
|||
## Current cookbooks |
|||
|
|||
- [nginx](nginx/): starts Nginx, content is initialized / updated from a GIT repository |
|||
- [postgresql](postgresql/): starts a PostgreSQL server, handles automated major upgrades, periodic backup and initialization of the database from the last backup. |
|||
|
|||
## License |
|||
|
|||
MIT |
|||
@ -1,2 +1,8 @@ |
|||
PARENT_DIR := .. |
|||
include $(PARENT_DIR)/Makefile |
|||
|
|||
.PHONY: test |
|||
|
|||
test: uninstall clean install |
|||
@echo "Running Nginx tests..." |
|||
curl -sSfL -I http://localhost/ |
|||
|
|||
@ -0,0 +1,60 @@ |
|||
# Podman Quadlet: Nginx |
|||
|
|||
## Overview |
|||
|
|||
Nginx is started as a Podman Quadlet and before that, the content to serve is initialized (`git clone`) or updated (`git pull`) from a GIT repository. |
|||
|
|||
## Usage |
|||
|
|||
In a separate terminal, follow the logs. |
|||
|
|||
```sh |
|||
sudo make tail-logs |
|||
``` |
|||
|
|||
Install the Podman Quadlets and start Nginx. |
|||
|
|||
```sh |
|||
sudo make clean install |
|||
``` |
|||
|
|||
You should see the **nginx-init.service** cloning this git repository to fetch the content to serve. |
|||
Then, the **nginx-server.service** should start up. |
|||
|
|||
You can check that the content is indeed served on port 80. |
|||
|
|||
``` |
|||
$ curl http://localhost/ |
|||
<!DOCTYPE html> |
|||
<html lang="en"> |
|||
<head> |
|||
<meta charset="UTF-8"> |
|||
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
|||
<title>Hello World</title> |
|||
</head> |
|||
<body> |
|||
<h1>Hello World</h1> |
|||
</body> |
|||
</html> |
|||
``` |
|||
|
|||
Then restart the **nginx.target** unit. |
|||
|
|||
```sh |
|||
sudo systemctl restart nginx.target |
|||
``` |
|||
|
|||
In the logs, you should see the **nginx-update.service** starting up and executing a `git pull` to update the content to serve. |
|||
Then, the **nginx-server.service** should start up. |
|||
|
|||
Finally, remove the quadlets, their configuration and their data. |
|||
|
|||
```sh |
|||
sudo make uninstall clean |
|||
``` |
|||
|
|||
## Integration tests |
|||
|
|||
```sh |
|||
sudo make test |
|||
``` |
|||
@ -1,2 +1,39 @@ |
|||
PARENT_DIR := .. |
|||
include $(PARENT_DIR)/Makefile |
|||
|
|||
.PHONY: test test-set-pgmajor |
|||
|
|||
PG_MAJOR_START ?= 14 |
|||
PG_MAJOR_LAST ?= 18 |
|||
test-set-pgmajor: |
|||
sed -i 's/^PG_MAJOR=.*/PG_MAJOR=$(PG_MAJOR_START)/' config/config.env |
|||
|
|||
test: uninstall clean test-set-pgmajor install |
|||
@echo "Running PostgreSQL integration tests..."; \
|
|||
set -Eeuo pipefail; \
|
|||
sleep 2; \
|
|||
echo "Creating a test database and a witness table..."; \
|
|||
podman exec postgresql-server su postgres -c "createdb test"; \
|
|||
podman exec postgresql-server su postgres -c "psql -U postgres -d test -c \"CREATE TABLE witness (id SERIAL PRIMARY KEY, version VARCHAR); INSERT INTO witness (version) SELECT version();\""; \
|
|||
podman exec postgresql-server su postgres -c "psql -U postgres -d test -c \"SELECT * FROM witness;\""; \
|
|||
for (( ver=$(PG_MAJOR_START); ver<$(PG_MAJOR_LAST); ver++ )); do \
|
|||
echo "Running a backup..."; \
|
|||
nextver=$$(($$ver + 1)); \
|
|||
systemctl start postgresql-backup.service; \
|
|||
systemctl stop postgresql.target; \
|
|||
sleep 1; \
|
|||
rm -rf /var/lib/quadlets/postgresql/{$$ver,$$nextver,data,latest,.initialized}; \
|
|||
echo "Restoring the backup to PostgreSQL $$ver..."; \
|
|||
systemctl start postgresql.target; \
|
|||
sleep 2; \
|
|||
podman exec postgresql-server su postgres -c "psql -U postgres -d test -c \"SELECT * FROM witness;\""; \
|
|||
echo "Testing upgrade from PostgreSQL $$ver to $$nextver..."; \
|
|||
systemctl stop postgresql.target; \
|
|||
sed -i "s/^PG_MAJOR=.*/PG_MAJOR=$$nextver/" /etc/quadlets/postgresql/config.env; \
|
|||
systemctl start postgresql.target; \
|
|||
sleep 2; \
|
|||
echo "Inserting line into the witness table..."; \
|
|||
podman exec postgresql-server su postgres -c "psql -U postgres -d test -c \"INSERT INTO witness (version) SELECT version();\""; \
|
|||
done; \
|
|||
podman exec postgresql-server su postgres -c "psql -U postgres -d test -c \"SELECT * FROM witness;\""; \
|
|||
echo "PostgreSQL upgrade tests completed." |
|||
|
|||
@ -0,0 +1,104 @@ |
|||
# Podman Quadlet: PostgreSQL |
|||
|
|||
## Overview |
|||
|
|||
PostgreSQL is started as a Podman Quadlet and before that, the database is initialized: |
|||
|
|||
- either from a previous backup, |
|||
- or from scratch using SQL statements or scripts. |
|||
|
|||
The upgrade process between major versions is handled by a one-off job before the database server startup. |
|||
|
|||
Finally, a Podman Quadlet is provided to perform a backup of the database, including a simple retention policy. |
|||
|
|||
## Usage |
|||
|
|||
In a separate terminal, follow the logs. |
|||
|
|||
```sh |
|||
sudo make tail-logs |
|||
``` |
|||
|
|||
Install the Podman Quadlets and start PostgreSQL. |
|||
|
|||
```sh |
|||
sudo make clean install |
|||
``` |
|||
|
|||
You should see the **postgresql-set-major.service** starting up to set the symlink pointing to the PGDATA directory of the desired major version. |
|||
Then, the **postgresql-init.service** should start up and initialize the database from scratch. |
|||
Finally, the **postgresql-server.service** is started. |
|||
|
|||
Restart the **postgresql.target** unit. |
|||
|
|||
```sh |
|||
sudo systemctl restart postgresql.target |
|||
``` |
|||
|
|||
You should see in the logs that the **postgresql-init.service** is skipped (because the database is already initialized) and the **postgresql-server.service** unit is started. |
|||
|
|||
Increment the PostgreSQL major version number. |
|||
|
|||
```sh |
|||
awk -i inplace -F= '/PG_MAJOR=/ { $2=$2+1; print $1"="$2; next } 1' /etc/quadlets/postgresql/config.env |
|||
``` |
|||
|
|||
Restart the **postgresql.target** unit. |
|||
|
|||
```sh |
|||
sudo systemctl restart postgresql.target |
|||
``` |
|||
|
|||
In the logs, you should see that the **postgresql-upgrade.service** converts the database files to the new major version. |
|||
|
|||
Make backups of the database. |
|||
|
|||
```sh |
|||
for i in $(seq 1 10); do |
|||
sudo systemctl start postgresql-backup.service |
|||
sleep 1 |
|||
done |
|||
``` |
|||
|
|||
In the logs, you should see ten runs of the **postgresql-backup.service** unit. |
|||
And in the three last runs, the retention policy should be kicked in to clean up old backup files. |
|||
|
|||
Now, stop the database server. |
|||
|
|||
```sh |
|||
sudo systemctl stop postgresql.target |
|||
``` |
|||
|
|||
Remove all the PostgreSQL files (except the backups). |
|||
|
|||
```sh |
|||
sudo find /var/lib/quadlets/postgresql/ -maxdepth 1 -mindepth 1 \! -name backup -exec rm -rf '{}' \; |
|||
``` |
|||
|
|||
Confirm there is no more data in `/var/lib/quadlets/postgresql`. |
|||
|
|||
``` |
|||
$ sudo ls -l /var/lib/quadlets/postgresql |
|||
total 0 |
|||
drwx------. 1 avahi avahi 38 1 déc. 21:04 backup |
|||
``` |
|||
|
|||
Start the PostgreSQL database server. |
|||
|
|||
```sh |
|||
sudo systemctl start postgresql.target |
|||
``` |
|||
|
|||
In the logs, you should see the **postgresql-init.service** unit restoring the database from the last backup. |
|||
|
|||
Finally, remove the quadlets, their configuration and their data. |
|||
|
|||
```sh |
|||
sudo make uninstall clean |
|||
``` |
|||
|
|||
## Integration tests |
|||
|
|||
```sh |
|||
sudo make test |
|||
``` |
|||
@ -0,0 +1,149 @@ |
|||
#!/bin/bash |
|||
|
|||
set -Eeuo pipefail |
|||
|
|||
# Find the latest PostgreSQL data directory |
|||
SOURCE_PGDATA="" |
|||
last_version="" |
|||
for version_file in /var/lib/postgresql/*/docker/PG_VERSION; do |
|||
if [ -f "$version_file" ]; then |
|||
version_dir=$(dirname "$version_file") |
|||
version_major=$(cat "$version_file") |
|||
if [ -z "$last_version" ] || [ "$version_major" -gt "$last_version" ]; then |
|||
last_version="$version_major" |
|||
SOURCE_PGDATA="$version_dir" |
|||
fi |
|||
fi |
|||
done |
|||
if [ -z "$SOURCE_PGDATA" ] || [ ! -d "$SOURCE_PGDATA" ]; then |
|||
echo "No PostgreSQL data directory found." |
|||
exit 1 |
|||
fi |
|||
echo "Using PostgreSQL data directory: $SOURCE_PGDATA" |
|||
|
|||
# Upgrade destination |
|||
TARGET_MAJOR_VERSION=${PGTARGET%%.*} |
|||
TARGET_PATH="/usr/local/bin/" |
|||
TARGET_PGDATA="/var/lib/postgresql/${TARGET_MAJOR_VERSION}/docker" |
|||
|
|||
# Upgrade source |
|||
SOURCE_MAJOR_VERSION=$(cat "${SOURCE_PGDATA}/PG_VERSION") |
|||
SOURCE_PATH="/usr/local-pg${SOURCE_MAJOR_VERSION}/bin" |
|||
|
|||
# Reuse functions from the official entrypoint script |
|||
source /usr/local/bin/postgres-docker-entrypoint.sh |
|||
|
|||
# Because they may have been over by the sourced script, reset all flags |
|||
set -Eeuo pipefail |
|||
|
|||
# if first arg looks like a flag, assume we want to run postgres server |
|||
if [ "$#" -eq 0 ] || [ "${1:0:1}" = '-' ]; then |
|||
set -- postgres "$@" |
|||
fi |
|||
|
|||
# Setup environment variables |
|||
docker_setup_env |
|||
|
|||
## |
|||
## Sanity checks |
|||
## |
|||
|
|||
# No need to upgrade if same major version |
|||
if [ "${SOURCE_MAJOR_VERSION}" == "${TARGET_MAJOR_VERSION}" ]; then |
|||
echo "PostgreSQL data files version matches target version. No upgrade required." |
|||
exit 0 |
|||
fi |
|||
# No automatic upgrade support for PostgreSQL versions less than 14 |
|||
if [ "${SOURCE_MAJOR_VERSION}" -lt 14 ]; then |
|||
echo "PosgreSQL <14 is no longer supported for automatic upgrade. Please perform a manual upgrade." |
|||
exit 1 |
|||
fi |
|||
# No downgrade support |
|||
if [ "${SOURCE_MAJOR_VERSION}" -gt "${TARGET_MAJOR_VERSION}" ]; then |
|||
echo "Downgrades are not supported. Aborting." |
|||
exit 1 |
|||
fi |
|||
# Check for concurrent upgrade processes |
|||
if [ -f "${SOURCE_PGDATA}/upgrade_in_progress.lock" ]; then |
|||
echo "Another upgrade process seems to be running (upgrade_in_progress.lock file found). Aborting." |
|||
exit 2 |
|||
fi |
|||
|
|||
# On PG v18, we have to check that data checksums, be it positive or negative, is set on the initdb args |
|||
# even when the user already provided initdb args, because otherwise Postgres v18 assumes you want checksums |
|||
# we now do this on every version to avoid one more conditional |
|||
# it also adds support for people who used Postgres with checksums before v18 |
|||
if [[ -z "${POSTGRES_INITDB_ARGS:-}" || "${POSTGRES_INITDB_ARGS:-}" != *"data-checksums"* ]]; then |
|||
DATA_CHECKSUMS_ENABLED=$(echo 'SHOW DATA_CHECKSUMS' | "${SOURCE_PATH}/postgres" --single "${@:2}" -D "${SOURCE_PGDATA}" "${POSTGRES_DB}" | grep 'data_checksums = "' | cut -d '"' -f 2) |
|||
|
|||
if [ "$DATA_CHECKSUMS_ENABLED" == "on" ]; then |
|||
DATA_CHECKSUMS_PARAMETER="--data-checksums" |
|||
elif [ "$TARGET_MAJOR_VERSION" -eq 18 ]; then |
|||
# Postgres v18 enables data checksums by default and is the only version with this opt-out parameter |
|||
DATA_CHECKSUMS_PARAMETER="--no-data-checksums" |
|||
fi |
|||
POSTGRES_INITDB_ARGS="${POSTGRES_INITDB_ARGS:-} ${DATA_CHECKSUMS_PARAMETER:-}" |
|||
fi |
|||
|
|||
# Flags the data directory as being in the middle of an upgrade |
|||
mkdir -p "${TARGET_PGDATA}" |
|||
touch "${SOURCE_PGDATA}/upgrade_in_progress.lock" |
|||
|
|||
# Now PGDATA points to the target data directory |
|||
export PGDATA="${TARGET_PGDATA}" |
|||
|
|||
# Initialize target data directory |
|||
docker_verify_minimum_env |
|||
docker_init_database_dir |
|||
|
|||
# Change into the PostgreSQL database directory, to avoid a pg_upgrade error about write permissions |
|||
cd "${PGDATA}" |
|||
|
|||
# Perform the upgrade |
|||
echo "Upgrading PostgreSQL from version ${SOURCE_MAJOR_VERSION} to ${TARGET_MAJOR_VERSION}..." |
|||
"${TARGET_PATH}/pg_upgrade" --username="${POSTGRES_USER}" --link \ |
|||
--old-datadir "${SOURCE_PGDATA}" --new-datadir "${TARGET_PGDATA}" \ |
|||
--old-bindir "${SOURCE_PATH}" --new-bindir "${TARGET_PATH}" \ |
|||
--socketdir="/var/run/postgresql" \ |
|||
--old-options "${run_options[*]}" --new-options "${run_options[*]}" |
|||
|
|||
# Re-use the pg_hba.conf and pg_ident.conf from the old data directory |
|||
cp -f "${SOURCE_PGDATA}/pg_hba.conf" "${SOURCE_PGDATA}/pg_ident.conf" "${TARGET_PGDATA}" |
|||
|
|||
# Set PGPASSWORD in case password authentication is used |
|||
if [ -z "${PGPASSWORD:-}" ] && [ -n "${POSTGRES_PASSWORD:-}" ]; then |
|||
export PGPASSWORD="${POSTGRES_PASSWORD}" |
|||
fi |
|||
|
|||
# Start a temporary PostgreSQL server |
|||
docker_temp_server_start "$@" |
|||
|
|||
if [ -n "${POSTGRES_UPDATE_SCRIPT:-}" ] && [ -f "${POSTGRES_UPDATE_SCRIPT}" ]; then |
|||
echo "Running update script: ${POSTGRES_UPDATE_SCRIPT}" |
|||
psql --username="${POSTGRES_USER}" -f "${POSTGRES_UPDATE_SCRIPT}" |
|||
fi |
|||
|
|||
echo "Updating query planner stats" |
|||
declare -a database_list=( $(echo 'SELECT datname FROM pg_catalog.pg_database WHERE datistemplate IS FALSE' | psql --username="${POSTGRES_USER}" -1t --csv "${POSTGRES_DB}") ) |
|||
for database in "${database_list[@]}"; do |
|||
echo "VACUUM (ANALYZE, VERBOSE, INDEX_CLEANUP FALSE)" | psql --username="${POSTGRES_USER}" -t --csv "${database}" |
|||
done |
|||
|
|||
if [[ "${PGAUTO_REINDEX:-}" != "no" ]]; then |
|||
echo "Reindexing the databases" |
|||
|
|||
if [[ "$TARGET_MAJOR_VERSION" -le 15 ]]; then |
|||
reindexdb --all --username="${POSTGRES_USER}" |
|||
else |
|||
reindexdb --all --concurrently --username="${POSTGRES_USER}" |
|||
fi |
|||
echo "End of reindexing the databases" |
|||
fi |
|||
|
|||
# Stop the temporary PostgreSQL server |
|||
unset PGPASSWORD |
|||
docker_temp_server_stop |
|||
|
|||
# Clean up lock files |
|||
rm -f "${SOURCE_PGDATA}/upgrade_in_progress.lock" |
|||
echo "PostgreSQL upgrade from version ${SOURCE_MAJOR_VERSION} to ${TARGET_MAJOR_VERSION} completed successfully." |
|||
Loading…
Reference in new issue