Kafka to Postgresql v2

The technical documentation of the kafka-to-postgresql-v2 microservice, which consumes messages from a Kafka broker and writes them in a PostgreSQL database by following the UMH data model v2.

Kubernetes resources

  • Deployment: united-manufacturing-hub-kafkatopostgresqlv2

Configuration

You shouldn’t need to configure kafka-to-postgresql-v2 manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the kafkatopostgresqlv2 section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
KAFKA_BROKERSSpecifies the URLs and required ports of Kafka brokers using the Kafka protocol.stringAnyunited-manufacturing-hub-kafka:9092
KAFKA_HTTP_BROKERSSpecifies the URLs and required ports of Kafka brokers using the HTTP protocol.stringAnyunited-manufacturing-hub-kafka:8082
LOGGING_LEVELDetermines the verbosity of the logging output, primarily used for development purposes.stringPRODUCTION, DEVELOPMENTPRODUCTION
POSTGRES_DATABASEDesignates the name of the target PostgreSQL database.stringAnyumh_v2
POSTGRES_HOSTIdentifies the hostname for the PostgreSQL database server.stringAnyunited-manufacturing-hub
POSTGRES_LRU_CACHE_SIZEDetermines the size of the Least Recently Used (LRU) cache for asset ID storage. This cache is optimized for minimal memory usage.stringAny1000
POSTGRES_PASSWORDSets the password for accessing the PostgreSQL databasestringAnychangemetoo
POSTGRES_PORTSpecifies the network port for the PostgreSQL database server.stringAny5432
POSTGRES_SSL_MODEConfigures the PostgreSQL connection to use SSL if set to ’true'.stringAnyrequire
POSTGRES_USERDefines the username for PostgreSQL database access.stringAnykafkatopostgresqlv2
VALUE_CHANNEL_SIZESets the size of the channel for message storage prior to insertion. This parameter is significant for memory consumptionstringAny10000
WORKER_MULTIPLIERThis multiplier affects the number of workers converting Kafka messages into the PostgreSQL schema. Total workers = cores * multiplier.stringAny16
Last modified January 22, 2024: Update version (074cc23)